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ABSTRACT 


The  Naval  Air  Systems  Command  (NAVAIR)  systems  engineering  technical 
review  (SETR)  process  does  not  acknowledge  commercial  documentation  of  military 
commercial  derivative  aircraft  (CD A)  programs.  The  Federal  Aviation  Administration’s 
(FAA)  type  certificate  (TC)  process  is  for  airworthiness  only,  but  is  extensive.  This  thesis 
compares  the  NAVAIR  SETR  timeline  to  the  FAA  TC  process  by  phase.  The  majority  of 
the  FAA  TC  process  effort  occurs  in  the  implementation  phase,  which  maps  to  test 
readiness  review  (TRR),  flight  readiness  review  (FRR),  functional  configuration  audit 
(FCA),  and  system  verification  review  (SVR).  These  events  assess  requirement  verification 
planning  and  results,  inclusive  of  airworthiness  and  performance  requirements.  The 
products  of  this  thesis  are  diagrams  mapping  the  commercial  documents  of  the 
implementation  phase  to  the  SETR  entrance  criteria  of  these  events.  Out  of  30  evaluated 
SETR  entrance  criteria,  22  map  to  FAA  elements.  A  case  study  of  a  military  CDA  program, 
the  Presidential  Helicopter  Replacement  Program,  evaluates  the  timeline  comparison  and 
validates  applicability  of  the  mapping  diagrams.  Mapping  of  the  remaining  SETR  events 
would  enable  further  benefits.  A  reinstatement  and  rewrite  of  the  cancelled  NAVAIRINST 
13100.15,  Engineering  Technical  Review  of  Commercial-Derivative  Aircraft  Programs , 
would  restore  NAVAIR  policy  on  supporting  military  CDA  programs  and  their  efficiencies 
with  SETR. 
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EXECUTIVE  SUMMARY 


The  Under  Secretary  of  Defense  (AT&L)  directs  using  commercial  products  and 
processes  where  possible  to  reduce  cost  and  time  to  execute  military  acquisition 
programs  (2003).  Naval  Air  Systems  Command  (NAVAIR)  certifies  airworthiness  and 
evaluates  performance,  being  the  military  airworthiness  authority  and  the  procuring 
customer.  The  Federal  Aviation  Administration  (FAA)  only  certifies  civil  airworthiness, 
but  reviews  a  great  deal  of  performance  in  the  process  of  ensuring  performance  does  not 
negatively  affect  airworthiness  (U.S.  Department  of  Transportation  Federal  Aviation 
Administration  2007).  The  FAA  requires  full  compliance  to  its  safety  standards,  while 
NAVAIR  allows  levels  of  safety  risk  with  acceptance  at  increasing  authority  levels 
(Airworthiness  and  CYBERSAFE  Directorate  [AIR-4. OP]  2016).  This  provides  the 
opportunity  for  leveraging  NAVAIR’ s  acceptance  of  performance  requirements  through 
the  FAA’s  acceptance  of  airworthiness  requirements. 

The  NAVAIR  systems  engineering  technical  review  (SETR)  process  is  baselined 
to  a  full  developmental  military  program,  with  programs  tailoring  down  as  appropriate. 
There  is  no  acknowledgement  of  commercial  processes  or  documentation  in  the  NAVAIR 
SETR  process  document,  NAVAIRINST  4355.19  (2015).  The  NAVAIR  Airworthiness 
and  CYBERSAFE  Directorate  (AIR-4. OP)  explicitly  accepts  FAA  airworthiness 
certifications  without  additional  technical  review  (2016).  Accepting  already-executed  FAA 
processes  in  SETR  reviews  would  leverage  further  the  value  added  of  commercial 
derivative  aircraft  (CDA)  program  cost  efficiencies  by  avoiding  extraneous,  duplicative 
effort  during  program  execution.  The  benefit  to  NAVAIR  SETR  test  readiness  review 
(TRR),  flight  readiness  review  (FRR),  functional  configuration  audit  (FCA),  and  system 
verification  review  (SVR)  is  provided  by  mapping  FAA  processes  and  documentation  to 
SETR  entrance  criteria.  These  four  SETR  events  focus  on  requirement  verification 
planning  and  evaluation,  where  the  majority  of  the  FAA  type  certificate  (TC)  process  effort 
aligns  with  NAVAIR  SETR  efforts.  By  identifying  FAA  elements  that  map  to  NAVAIR 
SETR  entrance  criteria,  military  CDA  programs  can  reduce  SETR  and  program  execution 
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effort  by  acknowledging  and  accepting  the  FAA  elements  instead  of  requiring  duplicative 
effort  to  NAVAIR  standards. 

This  research  is  empirical,  examining  existing  documentation  and  guidance.  The 
primary  sources  of  data  are  NAVAIRINST  4355.19  for  the  SETR  process  and  FAA  Order 
8 1 10.4  for  the  TC  process.  NAVAIRINST  13 100. 15  is  a  cancelled  instruction  that  provided 
a  poor  comparison  of  the  FAA  TC  process  to  NAVAIR  technical  reviews;  this  instruction 
is  highly  relevant  to  this  thesis,  but  disappointing  in  its  execution  (2002).  Discussion  of 
requirement  definition  documents  for  NAVAIR  and  the  FAA  frames  the  difference  in  focus 
and  confusion  of  airworthiness  versus  performance  requirement  definitions.  The  remaining 
documents  discussed  are  pertinent  to  NAVAIR  and  FAA  airworthiness  definitions. 

A  side-by-side  comparison  of  the  NAVAIR  SETR  timeline  and  the  FAA  TC 
process  at  a  high  level  provides  a  direct  correlation  between  the  two.  This  overall  timeline 
comparison  is  then  broken  into  phases.  The  phase  comparisons  align  FAA  elements  to 
NAVAIR  SETR  events.  This  enables  the  creation  of  mapping  diagrams,  relating  FAA 
elements  to  specific  SETR  entrance  criteria.  The  final  products  of  this  thesis  are  mapping 
diagrams  for  TRR,  FRR,  FCA,  and  SVR.  While  the  FAA  elements  cannot  completely 
satisfy  any  given  entrance  criteria  due  to  NAVAIR  SETR  assessing  a  much  broader  scope, 
there  is  no  question  of  benefit.  For  TRR,  eight  out  of  eleven  entrance  criteria  mapped  to 
FAA  elements;  for  FRR,  seven  out  of  ten;  for  FCA,  four  out  of  five;  and  for  SVR,  three 
out  of  four.  Out  of  30  evaluated  SETR  entrance  criteria,  22  map  to  FAA  elements;  this  is 
over  73%.  This  high  ratio  proves  there  is  benefit  in  accepting  FAA  processes  and 
documentation  for  military  CDA  programs,  even  with  FAA  elements  providing  partial 
satisfaction  to  the  SETR  entrance  criteria. 

This  thesis  presents  a  use  case  study  on  the  Presidential  Helicopter  Replacement 
Program.  This  is  a  military  CDA  program  obtaining  an  FAA  airworthiness  certificate  to 
the  maximum  extent  possible,  inclusive  of  the  military  modifications  (NAVAIR  2016). 
Scoping  SETR  for  this  program  to  integration  of  new  and  modified  design  supports  the 
acquisition  strategy  of  accepting  the  baseline  performance  of  the  commercial  S-92A.  The 
NAVAIR  SETR  timeline  to  FAA  TC  process  comparison  for  this  program  changes  from 
the  original  analysis.  This  program  entered  at  Milestone  B,  requiring  the  FAA  TC  process 


to  fit  entirely  within  the  engineering  and  manufacturing  development  (EMD)  phase.  The 
NAVAIR  SETR  events  also  shift  to  accommodate  the  Milestone  B  entry.  This  maintains 
the  original  alignment  of  FAA  elements  to  SETR  events  and  the  correlation  remains 
unchanged.  The  SETR  entrance  criteria  mapping  diagrams  remain  valid  and  relevant  for 
this  program.  The  majority  of  requirements  for  this  program,  67%,  are  verifiable  by  the 
time  of  the  FAA  TC  release,  ensuring  the  SETR  entrance  criteria  mapping  provides  great 
potential  in  reducing  the  effort  of  evaluation. 

Recommendations  include  revitalizing  NAVAIRINST  13100.15,  and 
acknowledgement  of  FAA  elements  for  military  CDA  programs  within  NAVAIRINST 
4355.19.  Mapping  the  remainder  of  SETR  events  would  identify  additional  benefits. 
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I. 


INTRODUCTION 


The  differences  between  airworthiness  and  performance  requirements  affect 
requirements  development  and  requirements  verification.  Civil  aviation  and  military 
acquisition  approach  these  areas  differently.  In  the  case  of  a  military  commercial  derivative 
program,  the  military  technical  review  process  should  acknowledge  civil  processes  to 
leverage  additional  benefits.  This  chapter  also  describes  the  scope,  methodology,  and 
organization  of  this  thesis. 

A.  BACKGROUND 

The  evaluation  of  airworthiness  and  performance  requirements  is  often  confused 
and  blurred;  the  focus  is  different,  yet  there  can  be  overlap  (see  Figure  1).  The  relative  size 
and  overlap  area  of  the  airworthiness  and  performance  sets  will  vary  depending  on  the 
program.  Will  the  system  be  safe  to  fly?  How  well  will  it  perform?  These  are  two  very 
different  questions.  For  example,  a  radio’s  transmissions  must  not  cause  interference  with 
the  flight  control  computer’s  operations,  but  whether  that  radio  transmits  at  the  minimum 
required  range  is  a  separate  determination.  The  evaluation  of  a  requirement  may  address 
airworthiness  or  performance,  or  both  at  the  same  time. 


Figure  1.  Requirement  Venn  Diagram 

Requirements  development  for  a  program  considers  both  airworthiness  and 
performance.  Performance  requirements  derive  from  the  mission  requirements  that  drive 

1 


the  need  for  the  program’s  creation.  Airworthiness  requirements  ensure  an  aircraft  is 
capable  of  safe  take-off,  flight  and  landing;  the  safety-of-flight  (SOF)  assessment 
determines  the  associated  levels  of  safety  risk  (Airworthiness  and  CYBERSAFE 
Directorate  [AIR-4.0P]  2016).  When  evaluating  an  aircraft  design’s  compliance  to 
requirements,  performance  verification  often  requires  flight-testing;  but  an  aircraft  must  be 
safe  to  fly  prior  to  allowing  that  flight  test  to  occur.  Assessments  then  occur  in  phases,  first 
verifying  airworthiness  in  order  to  execute  flight  test,  then  evaluating  performance 
requirements.  Flight  tests  may  occur  in  incremental  stages,  with  reduced  limitations 
imposed  until  the  final  collection  of  sufficient  data  to  validate  safe  expansion  to  the  full 
requested  performance. 

The  Department  of  Defense  (DOD)  has  been  shifting  to  use  of  commercial  products 
and  processes  where  possible,  to  reduce  costs  and  time  needed  to  acquire  products  (Under 
Secretary  of  Defense  [AT&L]  2003).  Commercial  derivative  aircraft  (CDA)  programs  can 
provide  a  great  savings  in  both,  when  it  is  possible  to  use  a  commercial  platform  in  a 
military  application  with  minimized  modifications.  Implementing  systemic  change  to 
support  this  type  of  program  requires  a  great  deal  of  effort  and  is  an  on-going  effort  (Lucka 
2003).  Today,  the  Naval  Air  Systems  Command  (NAVAIR)  Systems  Engineering 
Technical  Review  (SETR)  process  does  not  take  advantage  of  already-exercised  processes 
for  commercial  derivative  programs,  requiring  a  full  and  comprehensive  assessment 
against  NAVAIR  standards. 

NAVAIR  evaluates  both  airworthiness  and  performance,  being  both  the 
airworthiness  authority  and  the  customer;  the  Federal  Aviation  Administration  (FAA)  only 
certifies  airworthiness,  leaving  performance  acceptance  to  the  commercial  customer  (U.S. 
Department  of  Transportation  Federal  Aviation  Administration  2007).  Nevertheless,  the 
FAA  assesses  all  performance  requirements  for  potential  to  affect  airworthiness,  and 
assesses  airworthiness  with  the  system  operating  as  intended.  As  a  result,  the  FAA 
evaluates  most  performance  requirements  as  part  of  the  airworthiness  certification.  The 
FAA’s  charter,  defined  by  49  U.S.C.  §  44701,  is  to  “promote  safe  flight  of  civil  aircraft  in 
air  commerce”  by  enforcing  regulations  defining  “minimum  standards  required  in  the 
interest  of  safety,”  aircraft  must  meet  these  regulations,  with  no  increase  in  safety  risk 
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tolerated.  NAV AIR’s  military  airworthiness  and  SOF  requirements  focus  on  the  integrity 
of  the  aircraft  itself  and  allow  for  increasing  levels  of  safety  risk,  with  acceptance  at 
increasing  authority  levels  (Airworthiness  and  CYBERSAFE  Directorate  [AIR-4.0P] 
2016).  These  different  approaches  result  in  NAV AIR  having  a  narrower  scope  of 
airworthiness  requirements  in  comparison  to  the  FAA,  as  shown  in  Figure  2.  Therefore, 
the  FAA  has  a  high  likelihood  of  assessing  what  NAV AIR  would  consider  a  performance- 
only  requirement  as  part  of  the  FAA  airworthiness  assessment.  The  remaining  performance 
requirements  are  the  few  the  FAA  would  not  consider  affecting  airworthiness  and  those  the 
FAA  cannot  certify  for  pure  military,  non-civil  applications. 


NAVAIR 

Performance 

Airworthiness 

FAA 

Airworthiness 


Figure  2.  NAV AIR  vs.  FAA  Requirements  Venn  Diagram 


B.  PROBLEM 

The  NAV AIR  SETR  process  does  not  take  full  advantage  of  potential  efficiencies 
by  accepting  existing  commercial  FAA  certifications  and  processes;  acknowledgement  and 
referencing  of  these  processes  and  relevant  documents  is  completely  lacking.  The 
NAV AIR  SETR  process  establishes  event-driven,  independent  technical  reviews,  defining 
entrance  criteria,  documentation  required,  and  questions  to  evaluate  technical  readiness  for 
the  event  (NAV AIR  2015b).  A  full  developmental  military  program  is  the  baseline,  with 
programs  tailoring  the  SETR  event  as  appropriate. 
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The  Department  of  Navy  (DON)  Airworthiness  and  CYBERSAFE  Directorate 
(AIR-4.0P)  does  acknowledge  commercial  processes  to  leverage  an  airworthiness 
assessment  (2016).  The  SETR  process  does  not  reference  the  airworthiness  process  even 
though  airworthiness  products  are  among  SETR  event  entrance  criteria.  While  NAVAIR 
executes  both  SETR  and  airworthiness  processes  according  to  these  instructions,  there  is  a 
missing  link  between  them;  such  a  linkage  would  open  the  door  to  a  common  acceptance 
of  commercial  processes.  The  SETR  process  could  gain  additional  benefits  in  requirements 
verification  planning  and  evaluation  assessments.  Understanding  and  acceptance  of 
already-executed  FAA  processes  would  help  to  leverage  further  the  value  added  of  CDA 
program  cost  efficiencies  by  avoiding  extraneous,  duplicative  effort  during  the  SETR 
process.  This  thesis  investigates  the  following  research  questions: 

•  What  is  the  benefit  to  NAVAIR  SETR  execution  on  CDA  programs  by 
considering  FAA  processes  and  documentation? 

•  For  requirements  verification  planning,  how  do  FAA  processes  and 
documentation  for  test  readiness  and  flight  readiness  map  to  NAVAIR 
SETR  Test  Readiness  Review  (TRR)  and  Flight  Readiness  Review  (ERR) 
entrance  criteria? 

•  For  requirements  verification  acceptance,  how  do  FAA  performance 
evaluations  map  to  NAVAIR  SETR  Functional  Configuration  Audit  (FCA) 
and  System  Verification  Review  (SVR)  entrance  criteria? 

There  are  additional  SETR  events  that  would  also  benefit,  but  this  thesis  will  remain 
focused  on  these  four  events.  The  majority  of  efficiency  will  be  here. 

Conducting  the  SETR  process  takes  a  great  deal  of  effort,  and  tailoring  to  a  unique 
program  is  difficult  and  subjective,  with  criteria  written  to  a  full-development  Government 
program  and  no  references  to  commercial  equivalent  terminology  or  processes.  By 
evaluating  the  similarities  and  differences,  this  thesis  provides  a  comparison  of  NAVAIR 
and  FAA  processes,  providing  guidance  on  reducing  effort  and  potentially  program  cost 
for  TRR,  ERR,  FCA,  and  SVR. 

C.  SCOPE  AND  METHODOLOGY 

This  thesis  focuses  on  manned  aircraft  typical  for  a  military  commercial  derivative 

program.  Balloons,  light-sport  aircraft,  amateur-built  aircraft  and  similar  are  under  the 

4 


purview  of  the  FAA,  but  not  discussed  within  this  thesis.  Unmanned  aviation  systems 
(UASs)  bring  their  own  additional  unique  considerations,  with  multiple  levels  of  criteria 
based  on  the  complexity  of  the  system  and  physical  size  of  the  vehicle.  Assessing  the 
additional  ramifications  of  UAS  programs  is  also  beyond  the  scope  of  this  thesis. 

The  research  is  empirical,  examining  existing  documentation  and  guidance.  Each 
document  review  determines  how  the  FAA  and  NAVAIR  cover  the  material.  A  side-by- 
side  layout  of  process  flow,  at  an  overall  high  level,  and  then  broken  down  to  segmented 
lower  levels,  provides  a  direct  correlation  between  NAVAIR  and  FAA  milestones.  This 
correlation  leads  to  SETR  event  specific  diagrams,  and  a  mapping  of  TRR,  FRR,  FCA  and 
SVR  entrance  criteria  to  FAA  processes  and  documentation  (Figure  3).  A  pertinent  case 
study,  the  new  Presidential  Helicopter  Replacement  Program,  determines  the  value  of  the 
benefits  identified. 
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NAVAIR 


FAA 


Figure  3.  SETR  Entrance  Criteria  to  FAA  Element  Mapping  Product 


D.  ORGANIZATION  OF  THESIS 

This  thesis  is  divided  into  five  chapters,  opening  with  an  introduction,  background, 
objectives,  scope,  methodology,  and  process  discussion.  The  document  review  is 
fundamental  to  the  research  of  this  paper;  key  documents  are  discussed,  providing  the  raw 
material  for  the  analysis.  The  analysis  chapter  then  compares  NAVAIR  data  to  FAA  data: 
airworthiness  similarities/differences,  process  timeline  comparisons  from  a  high  level  to  a 
low  level  are  the  key  tool  in  enabling  the  mapping  correlation,  and  then  the  mapping 
correlations  for  the  TRR,  ERR,  FCA,  and  SVR  SETR  events.  The  fourth  chapter  discusses 
a  case  study,  the  Presidential  Helicopter  Replacement  Program,  presenting  lessons  learned 
and  assessing  the  benefits  of  the  mapping  diagrams  previously  presented.  The  final  chapter 

contains  the  conclusions  and  recommends  areas  for  further  research. 
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II.  DOCUMENTATION  REVIEW 


A  research  review  found  only  one  relevant  piece  of  existing  research  directly 
applicable  to  the  topic  of  this  thesis,  a  previous  thesis  on  improving  the  NAVAIR  flight 
clearance  process  for  CDA  programs  (Lucka  2003).  No  other  published  literature  or  theses 
reviewed  discuss  SETR  and  commercial  processes,  or  airworthiness  in  comparison  to 
performance  assessments.  Examination  of  the  existing  documents  for  NAVAIR  and  the 
FAA  pertinent  to  the  processes  in  question  remains  the  best  option  for  review. 

This  chapter  has  two  major  sections,  the  first  on  NAVAIR  documentation,  and  the 
second  on  FAA  documentation.  Each  section  discusses  key  documents  and  presents  key 
points  and  data  for  later  analysis. 

A.  EXISTING  PROCESSES,  NAVAIR 

As  the  focus  of  this  thesis  is  to  determine  benefits  to  NAVAIR  processes,  this 
chapter  discusses  relevant  NAVAIR  documents  first.  Figure  4  provides  a  visual  of  the 
documents  reviewed,  categorized  as  requirement  or  procedural  defining. 


Requirements 


MIL-HDBK-516 

Joint  Service 

Airworthiness  Certification  Criteria 

Specification  Guides 

NAVAIRINST  13100.15 
Engineering  Technical  Review 
of  Commercial-Derivative 
Aircraft  Programs 


NAVAIRINST  4355.19 

NAVAIRINST  13034.1 

Systems  Engineering  Technical 

Flight  Clearance  Policy  for  Air 

Review  Process 

Vehicles  and  Aircraft  Systems 

Procedures 

Figure  4.  NAVAIR  Documentation 
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1.  NAVAIRINST  13100.15,  Engineering  Technical  Review  of 
Commercial-Derivative  Aircraft  Programs 

The  subject  of  this  instruction  directly  relates  to  the  topic  of  this  thesis,  leveraging 
commercial  process  for  engineering  technical  reviews.  This  instruction  directs  the 
leveraging  of  commercial  processes  in  a  CDA  program.  A  review  of  the  content  reveals  it 
is  lacking  in  specific  guidance  and  discussion  of  commercial  processes  as  well  as  their 
correlation  to  NAVAIR  reviews. 

This  document  creates  and  defines  “program  points”  at  critical  junctures  in  a  CDA 
program’s  execution  (NAVAIR  2002).  These  points  correlate  to  acquisition  development 
milestones,  but  are  unique  terminology  to  this  instruction.  Figure  5  graphically  compares 
these  program  points  to  two  SETR  events,  Preliminary  Design  Review  (PDR)  and  Critical 
Design  Review  (CDR),  and  the  FAA  Type  Certificate  (TC)/Supplemental  Type  Certificate 
(STC)  process.  While  specific  FAA  forms  are  referenced  within  this  figure,  no  FAA 
documents  are  cited  anywhere  in  the  instruction,  leaving  the  reader  completely  without 
references  to  full  details. 
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Sample  TC/STC  Process 


(8)  Initial  Operating 
Capability  (IOC) 
nWNATOPS/ 
Maintenance  Plan 


If  Flight  Test  not  required,  TLA  will  not  be  issued.  Project 
Coordinator  should  submit  Installation  Confoimity 
Inspection  Request  and  Engineering  Drawings 


Prior  to  STC  Approval,  applicant  Project  Coordinator  should 
compile  DER  approvals,  updated  System  Specification, 
document  on  FAA  Form  8 1 10-3  signed  by  authorized  DER|  any 
changes  to  previously  submitted  data  (note  revision  level 
update  Master  Drawing  List)  to  FAA  Project  Manager, 

Issue  STC  Approval  or  Amend  original  TC 


Figure  5.  New  Aircraft  Acquisition  or  Major  Modifications  Program  Point  Process  Flow. 

Source:  NAY  AIR  (2002). 


The  NAVAIR  airworthiness  instruction  references  NAVAIRINST  13100.15 
(NAVAIR  2016a).  This  referencing  is  inaccurate  though,  as  within  the  past  few  years  a 
cancellation  of  NAVAIRINST  13100.15  occurred  with  no  replacement  or  explanation. 
Lucka  was  hopeful  the  policy  in  this  instruction  would  help  to  initiate  a  paradigm  shift,  but 
he  noted  its  lack  of  detail  and  specifics  for  execution  (Lucka  2003).  The  creation  of  terms 
not  used  elsewhere  in  acquisition  or  product  development  in  conjunction  with  the  vague 

and  short  discussion  of  alignment  to  incompletely  referenced  commercial  processes,  lead 
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to  a  crippling  ability  of  this  instruction  being  useful.  In  spite  of  its  unique  nature  and  open 
pursuit  of  supporting  commercial  processes,  these  flaws  potentially  led  to  its  lack  of  use 
and  ultimate  demise. 

2.  NAVAIRINST  4355.19,  Systems  Engineering  Technical  Review  Process 

This  document  is  the  defining  instruction  for  the  NAVAIR  SETR  process. 
Independent  boards  assess  a  program’s  technical  status  for  development,  design  maturity, 
and  risks  (NAVAIR  2015b).  This  instruction  discusses  the  tailoring  process,  minimum 
recommended  reviews,  and  the  requirement  to  complete  a  checklist  for  each  review.  An 
enclosure  provides  each  review’s  entrance  criteria.  A  separate  handbook  provides 
additional  guidance  on  review  descriptions,  suggested  agenda,  and  timing  (NAVAIR 
2015a). 

There  are  four  reviews  evaluating  requirements  verification  planning  and 
compliance  assessment:  TRR,  ERR,  FCA,  and  SVR  (NAVAIR  2015a).  TRRs  focus  on 
ensuring  the  design  baseline  is  stable,  requirements  traced  to  test  planning  efforts,  and 
resource  availability  needed  to  conduct  testing.  FRRs  assess  similar  elements  as  a  TRR, 
but  are  specific  to  preparing  for  flight  in  support  of  testing  and  include  a  check  on 
airworthiness  and  flight  clearance  status.  As  a  data  audit  and  not  a  review  event,  FCAs 
validate  the  design  satisfies  the  functional  requirements.  Often  in  conjunction  with  an  FCA, 
an  SVR  provides  the  review  structure  for  the  audit  results  as  well  as  any  associated  risk 
assessments  for  incomplete  or  non-compliant  findings.  See  Figure  6  for  the  graphic  of  the 
SETR  timeline  to  see  the  relationship  of  these  events  to  each  other  and  the  entire  acquisition 
timeline  (NAVAIR  2015b). 
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Figure  6.  NAVAIR  System  Engineering  Technical  Review  Timeline. 

Adapted  from  NAVAIR  (2015b). 
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The  potential  for  leveraging  commercial  or  FAA  processes  is  discussed  nowhere  in 
the  instruction  nor  the  handbook.  There  are  no  explicit  considerations  or  provisions  for 
CD  A  programs. 

3.  NAVAIRINST  13034.1 ,  Airworthiness  and  Cybersecurity  Safety 
Policies  for  Air  Vehicles  and  Aircraft  Systems 

This  document  defines  the  policy  and  provides  instructions  for  executing  and 
managing  airworthiness  of  NAVAIR  programs  (NAVAIR  2016a).  Two  definitions  are 
critical  to  this  instruction  and  this  thesis: 

Assessment  of  the  airworthiness  of  an  air  system  configuration  determines 
its  ability  to  safely  attain,  sustain  and  terminate  flight  (“complete”  in  case 
of  UAS)  per  approved  usage  limits.  ...  Assessment  of  Safety  of  Flight 
(SOF)  determines  the  property  of  an  air  system  configuration  to  safely 
attain,  sustain,  and  terminate  or  complete  flight  (to  include  in-flight  or  post¬ 
flight  aircrew  survivability),  within  prescribed  and  accepted  limits  for  injury 
or  death  to  personnel,  damage  to  equipment,  property,  and/or  environment. 

The  intent  of  assessing  SOF  is  to  show  that  the  level  of  system  safety  risk 
(hazards  resulting  in  death,  injury,  occupational  illness,  damage  to  or  loss 
of  equipment  or  property,  or  damage  to  the  environment)  have  been 
appropriately  identified  by  the  Technical  Area  Experts  (TAE),  accepted  by 
the  appropriate  authority,  and  concurred  with  by  the  fleet  or  test  user  for 
high  and  serious  risks.  (Airworthiness  and  CYBERSAFE  Directorate  [AIR- 
4. OP]  2016,  1-2,  1-3) 

The  instruction  defines  six  tenets  key  to  establishing  and  maintaining  airworthiness: 

1.  Continuing  Airworthiness:  ensure  sustainment  of  airworthiness  through 
the  entire  life  cycle. 

a.  Configuration  Management:  ensure  processes  are  in  place  to  manage 
and  control  changes,  assessing  airworthiness  impacts  of  those  changes. 

b.  Maintenance:  ensure  procedures  are  in  place  for  repairs  and  sustainment 
of  airworthiness. 

c.  Material  Management:  ensure  processes  are  in  place  to  authorize 
manufacturers,  screen  for  counterfeits,  and  impose  extra  rigor  on  parts 
affecting  safety. 

2.  Training  Systems/Devices:  align  curricula  and  simulators  with  the  aircraft 
configuration  to  ensure  proper  training  of  aircrew  and  maintainers. 
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3.  Safety  Management:  assess  safety  risks,  mitigate  where  possible, 
document  and  obtain  acceptance  when  risk  level  is  above  normal  for  the 
mission. 

4.  Flight  Tests:  establish  incremental  envelope  expansion  criteria,  collecting 
data  by  flight  test. 

5.  SETR:  include  a  process  check  to  ensure  programmatic  pressures  do  not 
negatively  affect  airworthiness. 

6.  Airworthiness  Certification  Products:  release  documentation  certifying  an 
airworthiness  and  SOF  assessment. 

The  last  tenet,  airworthiness  certification  products,  is  an  assurance  of  a  thorough, 
independent  technical  review  for  both  airworthiness  and  SOF,  as  well  as  the  identification 
and  acceptance  of  safety  risks.  The  interim  flight  clearance  (IFC)  is  temporary,  for  a 
specific  configuration  and  with  a  defined  condition  and/or  date  for  expiration.  An  IFC  may 
or  may  not  flow  into  a  permanent  flight  clearance  (PFC),  a  configuration  controlled  and 
regularly  updated  operator’s  manual. 

This  instruction  explicitly  acknowledges  and  recognizes  airworthiness 
certifications  issued  by  the  FAA,  and  in  fact  directs  full  acceptance  of  these  certifications; 
no  engineering  review  of  the  FAA’s  work  or  supporting  documentation  is  required.  For 
CDA,  basing  NAVAIR  IFCs  and  PFCs  entirely  on  FAA  Type  Inspection  Authorizations 
(TIAs),  TCs,  or  STCs  is  acceptable.  Directing  acceptance  of  these  FAA  certifications  is  a 
significant  change  in  the  desired  direction  since  the  time  of  Fucka’s  thesis,  when  this 
instruction  only  recommended  use  of  FAA  data,  which  NAVAIR  engineering  did  not 
understand  nor  support  (Fucka  2003).  Understanding  the  application  of  these  certifications 
to  the  NAVAIR  program  is  fundamental,  and  NAVAIR  must  assess  any  modifications 
outside  of  or  impactful  to  the  FAA  certification. 

This  instruction  provides  the  overall  framework  for  NAVAIR’ s  airworthiness 
definitions,  processes,  concerns,  and  responsibilities.  Its  acknowledgement  and  direct 
acceptance  of  FAA  airworthiness  products  benefits  CDA  programs. 
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4.  MIL-HDBK-516  Department  of  Defense  Handbook,  Airworthiness 
Certification  Criteria 

By  subsystem  or  discipline,  this  handbook  defines  explicit  airworthiness 
certification  criteria,  identifying  standards  and  methods  of  compliance  for  each 
(Department  of  Defense  2014).  These  criteria  create  the  airworthiness  certification  basis 
for  each  military  service’s  airworthiness  authority.  The  handbook  allows  for  tailoring, 
whether  additive  or  subtractive,  and  discusses  methods  to  document  such  tailoring.  CDA 
programs  and  FAA  airworthiness  certification  is  explicitly  acknowledged  and  discussed  to 
a  limited  extent. 

NAVAIRINST  13034.1  references  this  handbook  in  several  places:  a  policy 
paragraph  recommending  the  development  of  an  airworthiness  qualification  plan  (AQP), 
stating  MIL-HDBK-516  “should”  be  used;  in  the  responsibilities  chapter,  directing 
integration  of  this  handbook  into  airworthiness  requirements;  and  in  the  chapter  discussing 
flight  operating  limitations,  as  a  reference  providing  detailed  data  requirements  (NAVAIR 
2016a).  In  FAA  Order  8110.101,  the  FAA  states  modifications  for  military  commercial 
derivative  aircraft  (MCDA)  that  cannot  meet  civil  criteria  must  meet  the  criteria  in  this 
handbook,  further  discussing  its  definitive  use  by  the  military  (2015). 

5.  Joint  Service  Specification  Guides  (JSSGs) 

The  DOD  uses  the  suite  of  JSSG  documents  as  a  basis  to  develop  performance 
requirements  for  aircraft  (Department  of  Defense  2004).  These  documents  provide  a 
template  for  individual  requirement  language,  rationale  to  justify  the  requirement, 
verification  considerations,  and  lessons  learned  from  past  programs.  The  JSSGs  do  not  use 
the  word  “airworthiness,”  or  reference  MIL-HDBK-516.  The  only  assessment  involving 
safety  for  flight  readiness  is  identification  of  safety  hazards  in  the  context  of  system  safety, 
not  airworthiness  (Department  of  Defense  2004). 

The  seam  between  airworthiness  and  performance  requirement  development  could 
greatly  benefit  by  an  open  acknowledgement  and  cross-referencing,  avoiding  potential 
overlap  or  conflicting  requirements.  Requirements  developed  from  the  JSSGs  could  also 
affect  airworthiness  or  SOF. 
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B.  EXISTING  PROCESSES,  FAA 

This  section  discusses  FAA  documents  relevant  to  this  thesis.  Figure  7  provides  a 
visual  of  these  documents,  categorized  as  regulation,  procedural  or  guidance.  This  section 
discusses  many  of  these  documents  in  general  and  not  in  specific;  showing  them  here 
provides  additional  context. 
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Regulations 


Code  of  Federal  Regulations  (CFR), 
Title  14  -  Aeronautics  and  Space, 
Parts  1  through  199 
Regulated  by  the  FAA 


Order  8130.2 

Order  8110.101 

Airworthiness 

Order  8110.4 

Type  Certification  for  Military 

Certification  of  Products 

Type  Certification 

Commercial  Derivative 

and  Articles 

Aircraft  (MCDA) 

Procedures  (to  follow  regulations) 


AC  20-169 

Guidance  for  Certification  of  Military  and 
Special  Mission  Modifications  and 
Equipment  for  Commercial  Derivative 
Aircraft  (CDA) 


Guidance  (on  the  "how") 


AC  20-152 

RTCA,  Inc.,  Document  RTCA/ 
DO-254,  Design  Assurance 
Guidance  for  Airborne 
Electronic  Flardware 

AC  20-115 

RTCA,  Inc.,  Document 
RTCA/DO-178B 

AC  21-16 

RTCA  Document  DO-160 
versions  D,  E  and  F, 
"Environmental  Conditions 
and  Test  Procedures  for 
Airborne  Equipment" 

AC  25- 10 

Guidance  for 
Installation  of  Misc. 
Non-Required  Electrical 
Equipment 

1  | 

RTCA/DO-254 

1  | 

RTCA/DO-178B 

1  | 

RTCA/DO-160 

Design  Assurance  Guidance 

Software  Considerations  in 

!  Environmental  Conditions 

for  Airborne  Electronic 

Airborne  Systems  and 

and  Test  Procedures  for 

Flardware 

Equipment  Certification 

Airborne  Equipment 

AC  20-168 

Certification  Guidance  for 
Installation  of  Non- 
Essential,  Non-Required 
Aircraft  Cabin  Systems  & 
Equipment 


RTCA/DO-313 
Certification  Guidance  for 
Installation  of  Non-Essential, 
Non-Required  Aircraft  Cabin 
Systems  &  Equipment 

L _ 


Figure  7.  FAA  Documentation 


16 


1. 


Civil  Document  Structure 


The  general  structuring  of  the  discussed  civil  documents  is  hierarchical.  The  Title 
14  Code  of  Federal  Regulations  (CFR)  is  law  and  contains  the  detailed  regulations  for 
minimum  aircraft  requirements.  FAA  Orders  provide  the  procedures  to  implement  the 
CFR.  The  FAA  also  issues  advisory  circulars  (ACs),  which  align  to  the  CFR,  and  provide 
guidance  on  how  to  show  compliance.  In  many  instances,  the  FAA  ACs  allow  use  of  Radio 
Technical  Commission  for  Aeronautics  (RTCA)  documentation  as  one  way  to  show 
compliance  to  the  specified  regulations  discussed  in  the  AC. 

2.  14  CFR 

Title  14  of  the  CFR  covers  aeronautics  and  space;  the  FAA  regulates  parts  1  through 
199.  Parts  1  through  49  address  aircraft,  and  parts  91  through  97  address  air  traffic  and 
general  operating  rules.  The  remaining  parts  address  topics  not  of  interest  to  this  thesis, 
such  as  air  carrier  operations,  airport  operations,  air  traffic  rules,  training  device 
qualification,  crewmember  certification,  and  administrative  procedures.  These  regulations 
define  the  minimum  standards  established  by  the  FAA  to  ensure  safe  flight  of  civil  aircraft, 
as  directed  by  49  U.S.C.  §  44701. 

3.  Order  8130.2,  Airworthiness  Certification  of  Products  and  Articles 

This  document  defines  the  policy  and  provides  instructions  for  executing  and 
managing  airworthiness  of  FAA  programs  (U.S.  Department  of  Transportation  Federal 
Aviation  Administration  2015a).  The  definition  of  airworthiness  with  two  criteria  is  critical 
to  this  instruction  and  this  thesis.  Meeting  both  conditions  is  required: 

The  aircraft  must  conform  to  its  type  design.  For  the  purpose  of 
airworthiness,  conformity  to  the  type  design  is  considered  attained  when  the 
aircraft  configuration  and  the  engine,  propeller,  and  articles  installed  are 
consistent  with  the  drawings,  specifications,  and  other  data  that  are  part  of 
the  type  certificate  (TC).  This  includes  any  supplemental  type  certificate 
(STC)  and  repairs  and  alterations  incorporated  into  the  aircraft. 

The  aircraft  must  be  in  a  condition  for  safe  operation.  This  refers  to  the 
condition  of  the  aircraft  relative  to  wear  and  deterioration,  for  example,  skin 
corrosion,  window  delamination/crazing,  fluid  leaks,  and  tire  wear.  (U.S. 
Department  of  Transportation  Order  8130.2  2015,  2-1) 
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This  order  defines  the  content  and  details  of  forms  for  airworthiness  applications 
and  certifications,  for  every  potential  type  of  FAA  program. 

4.  Order  8110.4,  Type  Certification 

This  document  defines  the  FAA  procedures  followed  to  issue  new  or  modified 
airworthiness  certifications,  identified  as  TCs.  These  are  inclusive  of  supplemental, 
amended,  and  amended  supplemental  TCs. 

Figures  8  and  9  show  the  flow  diagrams  for  the  type  certification  process  from  this 
order  (U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007).  This 
process  focuses  on  obtaining  and  maintaining  the  airworthiness  certificate.  It  describes  this 
process  as  broken  into  five  phases:  conceptual  design,  requirements  definition,  compliance 
planning,  implementation,  and  post  certification  activities.  Between  these  phases,  as  well 
as  before  certification  flight  tests,  the  applicant  and  the  FAA  hold  type  certification  board 
meetings  (TCBMs).  The  FAA  is  not  the  paying  customer  for  the  program,  and  is  concerned 
only  with  an  executable  schedule  supportable  by  the  FAA’s  own  resource  constraints.  Not 
passing  a  gate  review  may  greatly  delay  the  schedule  until  the  FAA  has  resources  available 
again. 
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Figure  8.  Typical  Type  Certification  Process.  Source:  U.S.  Department  of  Transportation  Federal  Aviation  Administration  (2007). 
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Model  of  the  Type  Certification  Process,  Implementation  Phase 


Figure  9.  Typical  Type  Certification  Process,  Implementation  Phase. 

Source:  U.S.  Department  of  Transportation  Federal  Aviation 
Administration  (2007). 


The  conceptual  design  phase  is  where  the  applicant  is  determining  the  bounds  of 
their  program  and  the  scope  of  a  new  TC.  During  this  phase,  the  applicant  can  ask  questions 
of  the  FAA  on  processes  in  general,  but  not  in  specific.  The  applicant  can  brief  the  FAA 
on  their  program,  giving  the  FAA  time  to  consider  resources  and  potential  levels  of 
oversight  engagement.  The  FAA  will  not  respond  in  specific  until  after  a  formal  TC 
application,  accompanied  by  at  least  a  draft  certification  plan  (CP). 

The  requirements  definition  phase  is  where  the  FAA  officially  acknowledges  the 
program,  and  makes  decisions  regarding  involvement  and  FAA  resource  availability, 
documented  in  the  certification  project  plan  (CPP).  The  applicant  begins  drafting  the 
project  specific  certification  plans  (PSCPs),  with  detailed  execution  planning.  Issue  papers 
discuss  potential  areas  of  compliance  risk  and  their  solutions.  The  FAA  also  defines  the 

certification  basis  in  this  phase,  specifying  which  14  CFR  criteria  are  pertinent. 
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The  compliance-planning  phase  finalizes  execution  detailed  planning.  An 
engineering  conformity  plan  establishes  quality  control  processes  and  conformity 
inspection  planning  efforts.  Completing  the  PSCPs  to  a  high  level  of  detail,  sufficient  to 
determine  a  high  likelihood  of  successful  execution,  is  required  to  complete  this  phase. 

The  implementation  phase  is  broken  down  further  into  three  sub-phases: 
compliance  data  generation,  compliance  substantiation,  and  compliance  finding.  The 
complexity  of  this  phase  led  to  its  separate  breakout  in  Figure  9.  In  the  compliance  data 
generation  sub-phase,  the  FAA  approves  test  plans  prior  to  execution.  The  applicant  can 
then  begin  component  level  testing,  sub-system  level  testing,  and  on-aircraft  testing,  as 
needed.  Performing  analyses,  building,  and  performing  compliance  inspections,  potentially 
witnessed  by  the  FAA,  all  happen  in  this  phase.  When  flight  test  is  required,  the  applicant 
obtains  an  experimental  airworthiness  certificate  from  the  FAA.  The  applicant  must 
successfully  validate  compliance  before  seeking  FAA  final  approval,  and  is  the  purpose  of 
this  sub-phase.  The  compliance  substantiation  sub-phase  is  where  the  applicant  submits  the 
data  to  the  FAA.  The  final  implementation  sub-phase  is  compliance  finding.  This  is  the 
phase  where  the  FAA  reviews  data  and  conducts  flight  tests.  This  is  the  FAA’s  review  of 
the  applicant’s  results  to  date,  and  authorizes  inspections  and  aircraft-level  tests  for  score. 
The  applicant  also  generates  instructions  for  continued  airworthiness,  also  known  as 
maintenance  manuals,  and  a  flight  manual. 

The  FAA  reviews  the  all  data,  analyses,  testing  results,  and  manuals.  If  the  FAA 
finds  compliance,  the  final  TCBM  results  in  the  FAA  issuing  a  TC,  with  a  complementary 
TC  data  sheet  (TCDS). 

The  post  certification  activities  phase  describes  airworthiness  maintenance 
activities  and  process  checks  on  change  impacts. 

The  remainder  of  this  order  discusses  process  points,  form  details,  unique 
certification  topics,  and  noise  certification. 
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5. 


Order  8110.101,  Type  Certification  Procedures  for  Military  Commercial 
Derivative  Aircraft 


This  document  discusses  how  the  FAA  evaluates  type  certification  for  MCDA 
(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2015b).  This  order  is 
a  supplement  to  Order  8110.4,  discussing  unique  issues  in  the  MCDA  context.  An 
agreement  between  the  military  and  the  FAA  resulted  in  the  creation  of  the  FAA’s  Military 
Certification  Office  (MCO).  The  MCO  manages  MCDA  airworthiness  projects, 
functioning  as  a  unique  entity  capable  of  understanding  military  applications  held  to  civil 
regulations.  Since  the  basic  premise  of  an  MCDA  is  to  begin  with  a  commercial  aircraft 
and  modify  it,  the  MCO  rarely  issues  new  TCs;  most  MCDA  projects  are  either  amended 
or  supplemental  type  certificates  (ATCs  or  STCs). 

A  note  in  this  document  highlights  that  the  procurement  contract  defines  the 
relationship  between  the  applicant  and  the  military;  the  contract  does  not  bind  the  FAA/ 
MCO.  While  the  MCO  exists  to  support  MCDA,  the  military  does  not  automatically 
receive  any  access  to  communications  between  the  FAA/MCO  and  the  applicant.  The 
contract  must  define  such. 

The  procedures  followed  are  still  per  Order  8110.4  for  obtaining  airworthiness 
certification.  Military  government  furnished  equipment  (GFE)  must  still  comply  with  civil 
regulations;  missing  data  is  either  gathered,  or  restrictions  are  imposed.  If  civil  regulations 
do  not  address  the  military  equipment  function,  it  is  not  eligible  for  an  FAA  airworthiness 
certificate,  such  as  weapons  or  electronic  jamming  systems.  In  cases  where  the  FAA  cannot 
test  the  performance  of  an  item  of  GFE,  such  as  encryption,  a  statement  from  the  military 
is  acceptable  attesting  to  the  item  meeting  its  intended  function.  This  order  also  discusses 
the  additional  points  of  process  regarding  funding,  military  sponsorship,  and  potential 
application  of  military  functions  to  civil  aviation. 

The  seams  between  the  FAA/MCO  and  the  military  must  be  clear.  The  FAA 
provides  a  list  of  what  deviations  are  not  certified.  The  military  assesses  these  deviations, 
in  addition  to  any  modifications  made  by  the  military  after  receipt,  for  airworthiness.  This 
order  also  touches  on  the  military  use  of  MIL-HDBK-516  for  determining  airworthiness 
criteria. 
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C.  CHAPTER  SUMMARY 

This  chapter  first  reviews  NAVAIR  documentation,  then  FAA  documentation.  The 
documents  reviewed  relate  to  each  entity’s  requirements  development  and  procedural 
guidance.  A  NAVAIR  instruction  on  technical  reviews  and  CDA  programs  is  directly  on 
point  for  this  thesis,  but  is  no  longer  active.  The  section  then  discusses  the  NAVAIR 
instructions  for  SETR  process  and  airworthiness  policy.  The  section  ends  with  descriptions 
of  two  documents  for  airworthiness  and  performance  requirements  development,  which  do 
not  reference  each  other.  The  chapter  then  switches  to  discussing  FAA  documentation, 
describing  the  structure  and  hierarchy  first.  The  section  briefly  describes  the  regulation 
defining  FAA  requirements,  then  the  order  defining  FAA  airworthiness.  A  description  of 
the  FAA  certification  process  order  follows,  with  an  additional  description  of  the  order 
created  to  support  military  CDA  in  the  FAA.  The  section  ends  with  a  short  overview  of  the 
unique  reviews  conducted  at  the  component  level.  The  analysis  in  the  next  chapter  uses  the 
data  and  information  provided  here. 
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III.  ANALYSIS  OF  DATA 


This  chapter  presents  the  analyses  of  the  data  discussed  in  the  previous  chapter.  A 
comparison  of  airworthiness  definitions  frames  the  difference  in  perspectives,  followed  by 
a  comparison  of  airworthiness  tenets  that  finds  common  ground.  Identification  of 
commonalities  between  TRR,  FRR,  FCA,  and  SVR  entrance  criteria  ensures  consistency 
in  the  final  analyses.  The  major  analysis  effort  compares  the  SETR  timeline  to  the  TC 
process.  This  begins  with  an  overall  simplified  view,  and  then  focuses  on  each  phase 
separately  to  identify  where  FAA  elements  align  with  SETR  events.  The  final  deliverables 
are  diagrams  mapping  identified  FAA  elements  to  specific  SETR  entrance  criteria  for  TRR, 
FRR,  FCA,  and  SVR. 

NAVAIR  and  the  FAA  have  fundamentally  different  approaches  for  documenting 
policies  and  procedures,  as  well  as  documentation  formatting  styles.  Even  internally  within 
NAVAIR  and  the  FAA,  different  documents’  structures  may  differ  greatly  given  different 
purposes,  authors,  and  historical  development.  Therefore,  different  methods  are 
appropriate  for  each  document  comparison.  The  empirical  analysis  comparison  types  are 
written,  tabular,  timeline  alignment,  and  mapping  diagrams.  The  differing  documents’ 
structures  and  intent  of  the  comparison  drive  the  format  selected  by  this  author. 

A.  AIRWORTHINESS  COMPARISONS 

This  section  will  discuss  the  similarities  and  differences  of  airworthiness 
approaches  between  NAVAIR  and  the  FAA.  The  analyses  justify  how  FAA  airworthiness 
assessment  scope  encompasses  NAVAIR’ s  in  the  majority,  and  why  NAVAIR-4.0P  allows 
use  of  FAA  airworthiness  products. 

1.  Airworthiness  Definitions 

Repeating  the  definitions  from  NAVAIR  and  the  FAA  allow  for  ready  comparison: 

Assessment  of  the  airworthiness  of  an  air  system  configuration  determines 
its  ability  to  safely  attain,  sustain  and  terminate  flight  (“complete”  in  case 
of  UAS)  per  approved  usage  limits.  ...  Assessment  of  Safety  of  Flight 
(SOF)  determines  the  property  of  an  air  system  configuration  to  safely 
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attain,  sustain,  and  terminate  or  complete  flight  (to  include  in-flight  or  post¬ 
flight  aircrew  survivability),  within  prescribed  and  accepted  limits  for  injury 
or  death  to  personnel,  damage  to  equipment,  property,  and/or  environment. 

The  intent  of  assessing  SOF  is  to  show  that  the  level  of  system  safety  risk 
(hazards  resulting  in  death,  injury,  occupational  illness,  damage  to  or  loss 
of  equipment  or  property,  or  damage  to  the  environment)  have  been 
appropriately  identified  by  the  Technical  Area  Experts  (TAE),  accepted  by 
the  appropriate  authority,  and  concurred  with  by  the  fleet  or  test  user  for 
high  and  serious  risks...  (Airworthiness  and  CYBERSAFE  Directorate 
[AIR-4.0P]  2016,  1-2,  1-3) 

The  aircraft  must  conform  to  its  type  design.  For  the  purpose  of 
airworthiness,  conformity  to  the  type  design  is  considered  attained  when  the 
aircraft  configuration  and  the  engine,  propeller,  and  articles  installed  are 
consistent  with  the  drawings,  specifications,  and  other  data  that  are  part  of 
the  type  certificate  (TC).  This  includes  any  supplemental  type  certificate 
(STC)  and  repairs  and  alterations  incorporated  into  the  aircraft. 

The  aircraft  must  be  in  a  condition  for  safe  operation.  This  refers  to  the 
condition  of  the  aircraft  relative  to  wear  and  deterioration,  for  example,  skin 
corrosion,  window  delamination/crazing,  fluid  leaks,  and  tire  wear.  (U.S. 
Department  of  Transportation  Federal  Aviation  Administration  2015a,  2-1) 

The  two  definitions  at  first  seem  fundamentally  different.  NAVAIR  focuses  first  on 
the  integrity  of  the  aircraft  itself  within  allowable  damage  limits  to  other  entities  and  then 
on  determination  of  safety  risk  level  and  acceptance  (Airworthiness  and  CYBERSAFE 
Directorate  [AIR-4. OP]  2016).  The  FAA  first  discusses  verifying  the  aircraft’s 
conformance  to  its  authorized  design  and  then  discusses  evaluating  the  condition  of  the 
aircraft  for  safe  operation  (U.S.  Department  of  Transportation  Federal  Aviation 
Administration  2015a). 

NAVAIR  pointedly  allows  for  varying  levels  of  safety  risk.  The  FAA’ s  requirement 
for  design  conformity  in  itself  allows  for  no  safety  risk  acceptance;  the  design  must  comply 
with  regulations,  the  minimum  requirements  for  safety.  It  follows  then,  the  FAA  evaluates 
requirements  NAVAIR  would  consider  performance  only.  This  difference  explains  the 
relative  size  of  the  airworthiness  requirement  bubbles  in  Figure  10.  The  NAVAIR  portion 
protruding  outside  of  the  FAA’s  is  for  areas  the  FAA  cannot  certify  due  to  unique  military 
applications. 
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Figure  10.  NAVAIR  vs.  FAA  Airworthiness  Venn  Diagram 

Unlike  the  FAA,  the  NAVAIR  airworthiness  definition  does  not  call  for  design 
conformity,  or  safe  maintenance.  NAV AIR-4. OP  addresses  these  topics  separately  in  the 
consideration  of  everything  impactful  to  airworthiness  (NAVAIR  2016a).  The  NAVAIR 
definition  itself  focuses  more  narrowly  than  the  FAA’s. 

At  first  glance,  the  definitions  seem  very  different,  but  do  have  a  great  deal  in 
common  when  framed  by  their  full  documentation.  Both  airworthiness  perspectives 
address  design  conformity  and  maintaining  in  a  safe  status.  Both  address  safety  risk, 
whether  explicitly  or  intrinsically;  they  differ  most  significantly  in  their  tolerance  of  safety 
risk. 


2.  Airworthiness  Tenets 

NAVAIR  lists  and  discusses  six  tenets  of  airworthiness  (NAVAIR  2016a).  While 
the  FAA  does  not  explicitly  identify  individual  tenets,  Table  1  identifies  equivalents  (U.S. 
Department  of  Transportation  Federal  Aviation  Administration  2007). 
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Table  1.  Airworthiness  Tenets  Comparison 


NAVAIR  13034.1  FAA 


1.  Continuing  Airworthiness 

la.  Configuration  Management 

Conformity  Inspection 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2015a) 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007) 

lb.  Maintenance 

Instructions  for  Continued  Airworthiness 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2015a) 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007) 

lc.  Material  Management 

Manufacturing  Production  Approval 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007) 

2.  Training 

Training 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2015a) 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007) 

3.  Safety 

14CFR 

4.  Flight  Tests 

Flight  Test 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2015a) 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007) 

5.  SETR 

TC  Process 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007) 

6.  Airworthiness  Products 

TC/STC 

(U.S.  Department  of  Transportation  Federal  Aviation  Administration  2007) 

While  using  different  terminologies,  both  entities  do  consider  the  same  tenets  as 
fundamental  to  establishing  and  maintaining  airworthiness.  Each  tenet  has  its  own  suite  of 
processes  for  management  and  detailed  implementation,  but  both  NAVAIR  and  the  FAA 
identify  these  key  areas  in  their  airworthiness  documentation.  This  similarity  enables  the 
airworthiness  leveraging  by  NAVAIR-4.0P. 
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B.  TRR,  FRR,  FCA,  SVR  SETR  ENTRANCE  CRITERIA  COMPARISONS 

This  section  analyses  NAVAIR  SETR  event  entrance  criteria  for  commonality, 
enabling  consistent  mapping  for  the  later  NAVAIR  SETR  timeline  to  FAA  TC  process 
comparison  analysis. 

The  entrance  criteria  for  TRR,  FRR,  FCA,  and  SVR  (NAVAIR  2015b)  have  a  great 
deal  in  common.  While  TRR  and  FRR  assess  test  readiness  and  flight  readiness, 
respectively,  FCA  and  SVR  are  often  performed  together  to  assess  requirement  compliance 
of  the  design.  Figure  1 1  maps  a  comparison  of  these  entrance  criteria;  a  solid  green  line 
indicates  duplicated  entrance  criteria,  a  double  green  line  indicates  related  entrance  criteria, 
and  a  lack  of  any  green  lines  indicates  a  unique  entrance  criteria.  See  the  Appendix  for  an 
excerpted  listing  of  the  full  entrance  criteria  descriptions  for  these  four  reviews. 
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Figure  11.  TRR,  FRR,  FCA,  and  SVR  Entrance  Criteria  Mapping 
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The  overlap  between  the  first  two  events  is  understandably  high;  a  TRR  may  or 
may  not  need  flight  to  gather  data,  while  an  FRR  focuses  specifically  on  flight  readiness. 
The  primary  overlap  across  all  four  events  is  the  entrance  criterion  assessing  requirements 
traceability  to  verification.  A  key  secondary  entrance  criterion  assesses  appropriate 
certifications  for  each  of  these  events. 

By  comparing  these  four  events  against  each  other  and  identifying  commonalities, 
Figure  1 1  provides  a  tool  to  ensure  the  FAA  element  comparison  analysis  performed  later 
in  this  chapter  is  consistent  in  mapping  to  common  and  related  entrance  criteria. 

C.  SETR  TO  TYPE  CERTIFICATION  PROCESS  COMPARISON 

This  section  presents  the  direct  mapping  analyses.  First,  this  section  compares  a 
simplistic  overall  SETR  timeline  to  the  overall  TC  process.  Separate  comparison  of  major 
phases  follow,  which  in  turn  enables  mapping  of  FAA  process  to  NAVAIR  SETR.  The 
final  products,  by  SETR  event,  map  FAA  elements  to  specific  entrance  criteria. 

1.  SETR  Timeline  to  FAA  Type  Certificate  Process  Comparison 

Not  all  phases  of  SETR  shown  in  Figure  6  are  necessary  for  this  thesis.  By  the 
nature  of  executing  a  CDA  program,  the  program  has  already  reached  many  critical 
decisions;  efforts  prior  to  Milestones  A  and  B  authorize  the  CDA  path.  Note  SETR  events 
occur  independent  of  what  milestone  or  phase  a  program  begins;  the  alignment  of  Figure 
6  is  notional  and  each  program  adjusts  as  needed  (NAVAIR  2015b).  The  FAA  TC  process 
focuses  only  on  obtaining  and  maintaining  the  airworthiness  certificate,  while  NAVAIR’ s 
SETR  process  supports  the  entire  acquisition  and  life  cycle  of  a  program.  Figure  12 
presents  a  comparison  of  a  simplified  NAVAIR  SETR  timeline  based  on  Figure  6  to  the 
FAA’s  TC  process  of  Figures  8  and  9. 
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Figure  12.  NAVAIR  SETR  to  FAA  TC  Process  Comparison 


32 


The  FAA’s  conceptual  design  phase  correlates  to  NAV AIR’s  material  solution  and 
analysis  phase.  This  initial  phase  scopes  the  program,  investigates  feasibility,  plans 
notional  execution  and  defines  overall  program  requirements.  Acquisition  execution 
dictates  a  shift  in  the  alignment  of  these  two  phases;  the  FAA  TC  process  would  not  begin 
until  after  selection  of  a  contractor  at  Milestone  A.  The  FAA’s  requirements  definition 
phase  correlates  to  NAV AIR’s  technology  maturation  and  risk  reduction  phase.  This  in- 
depth  phase  investigates  execution  feasibility  in  detail  and  finalizes  detailed  requirements. 
NAV AIR’s  engineering  and  manufacturing  development  (EMD)  phase  encompasses  two 
major  FAA  phases,  compliance  planning  and  implementation.  The  FAA’s  compliance¬ 
planning  phase  fully  develops  the  design;  in  the  NAV AIR  timeline,  PDR  and  CDR  assess 
the  design  to  full  maturity.  The  remainder  of  NAV AIR’s  EMD  phase  correlates  to  the 
FAA’s  implementation  phase.  The  implementation  phase  is  broken  down  further  into  three 
sub-phases:  data  generation,  substantiation  and  finding.  Build,  analysis,  and  testing  occur 
in  this  phase.  At  the  end  of  the  EMD  and  implementation  phases,  the  two  timelines 
converge  again,  with  the  final  assessment  determining  compliance.  NAV AIR  then  moves 
on  to  managing  production  execution  in  the  production  and  deployment  phase.  The  FAA 
TC  process  focuses  only  on  airworthiness  compliance,  so  the  production  phase  does  not 
have  a  counterpart;  this  is  an  aspect  of  NAV AIR  being  the  procuring  customer,  while  the 
FAA  is  not.  Finally,  the  FAA’s  post  certification  activities  phase  addresses  airworthiness 
maintenance  aspects.  NAV AIR’s  operations  and  support  phase  also  addresses  these,  in 
addition  to  many  other  considerations  relevant  to  being  the  user. 

a.  Conceptual  Design,  Requirements  Definition  Compared  to  Material 
Solution  and  Analysis,  Technology  Maturation  and  Risk  Reduction 
Phases 

Figure  13  focuses  on  the  FAA’s  conceptual  design  and  requirements  definition 
phases,  presenting  key  elements  for  comparison  to  the  NAV AIR  phases.  The  products  of 
the  conceptual  design  phase  and  true  kick-off  for  an  FAA  certification  project  are  the  initial 
CP  and  the  TC  application.  These  products  define  the  scope  of  the  program,  the  notional 
execution  schedule,  and  formal  request  by  the  applicant  to  the  FAA  to  initiate  the  TC 
process.  This  phase  has  the  same  purpose  as  NAV AIR’s  material  solution  and  design 
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analysis  phase  where  program  requirements  are  scoped  and  the  Milestone  A  decision 
authorizes  a  program  to  move  forward.  A  military  CDA  program  would  not  begin  the  FAA 
TC  process  until  after  selection  of  a  contractor,  so  the  FAA  conceptual  design  phase  would 
shift  to  the  right  as  shown  in  Figure  13.  The  applicant  submits  the  CP  and  TC  application 
after  showing  understanding  of  the  government’s  requirements  in  SRR-II.  The  preliminary 
TCBM  between  the  applicant  and  the  FAA  at  the  end  of  the  conceptual  design  phase  begins 
the  FAA’s  official  establishment  of  the  TC  effort. 
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Figure  13.  FAA  Key  Documents  for  First  and  Second  Phases 


The  products  of  the  requirements  definition  phase  are  the  initial  CPP,  issue  papers, 
initializing  PSCPs  and  the  certification  basis.  The  CPP  defines  execution  details  and 
resource  allocation  within  the  FAA;  issue  papers  describe  areas  of  technical  concern;  the 
initial  PSCPs  begin  to  frame  compliance  planning  details;  and  the  final  product  of  this 
phase  is  the  certification  basis,  defining  which  specific  regulations  must  be  satisfied.  This 
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phase  correlates  to  NAV AIR’s  technology  maturation  and  risk  reduction  phase  where 
investigation  determines  the  feasibility  of  execution  details  and  the  requirements  are 
refined  to  the  detailed  level.  As  draft  PSCPs  begin  to  flesh  out  the  design,  an  allocated 
baseline  emerges,  aligning  with  the  SFR.  The  definition  of  the  certification  basis  moves 
the  program  forward  into  the  next  phase  to  complete  design  work,  comparable  to  a 
Milestone  B  decision  authorizing  a  program  to  move  into  EMD.  The  FAA  and  applicant 
hold  an  interim  TCBM  for  certification  basis  at  the  end  of  the  requirements  definition 
phase. 


b.  Compliance  Planning  Compared  to  Engineering  and  Manufacturing 
Development  Phase 

Figure  14  focuses  on  the  FAA’s  compliance-planning  phase,  presenting  key 
elements  for  comparison  to  the  beginning  of  the  NAVAIR  EMD  phase.  The  products  of 
the  compliance-planning  phase  are  the  development  of  a  conformity  plan,  the  completion 
of  the  PSCPs,  and  a  finalized  certification  plan.  The  PSCPs  must  be  completed  and 
approved  by  the  FAA  in  order  to  move  into  the  implementation  phase  following.  These 
products  finalize  design  and  compliance  planning  details.  This  phase  has  the  same  purpose 
as  the  first  portion  of  NAVAIR’ s  EMD  phase,  fully  maturing  and  assessing  design  through 
PDR  and  CDR.  The  FAA  and  applicant  hold  an  interim  TCBM  for  certification  plan  at  the 
end  of  the  compliance -planning  phase. 
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Figure  14.  FAA  Key  Documents  for  Compliance  Planning  Phase 


c.  Implementation  Compared  to  Engineering  and  Manufacturing 
Development  Phase 

Figure  15  focuses  on  the  FAA’s  implementation  phase,  presenting  key  elements  in 
comparison  to  the  remainder  of  the  NAVAIR  EMD  phase.  This  is  the  execution  phase, 
where  analysis,  building,  and  testing  occur.  The  implementation  phase  is  further  broken 
down  into  three  sub-phases:  data  generation,  substantiation,  and  finding. 
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Figure  15.  FAA  Key  Documents  and  Processes  for  Implementation  Phase 


The  applicant  performs  a  great  deal  of  effort  in  the  data  generation  phase,  where 
analysis,  build,  and  testing  occur  to  the  applicant’s  satisfaction.  The  FAA  oversees  portions 
of  this  phase,  witnessing  conformity  inspections,  approving  all  test  plans,  and  providing  an 
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experimental  airworthiness  certificate  to  authorize  the  applicant’s  flight  tests.  This  phase 
correlates  to  NAV AIR’s  TRR,  where  assessment  of  the  design  evaluates  its  readiness  for 
component  or  sub-system  level  testing  as  appropriate.  TRRs  may  occur  several  times,  as 
needed  to  support  design  development  testing.  When  the  applicant  believes  all  compliance 
data  is  sufficient,  the  applicant  submits  the  data  to  the  FAA  in  the  substantiation  sub-phase. 

The  finding  sub-phase  is  critical,  as  this  is  where  the  FAA  will  evaluate  data  and 
perform  their  own  tests  for  score.  The  products  of  the  finding  sub-phase  are  the  TIA,  the 
flight-test-conformity  inspection,  instructions  for  continued  airworthiness,  and  the  flight 
manual.  The  TIA  formally  accepts  contractor  submitted  results  and  authorizes  the  FAA  to 
conduct  the  airworthiness  conformity  inspection  and  flight  tests  to  follow.  Concurrently, 
the  applicant  develops  and  submits  the  instructions  for  continued  airworthiness  (ICA) 
providing  maintenance  procedures  to  maintain  airworthiness,  and  the  flight  manual  to 
provide  flight  and  operational  procedures.  The  pre -flight  TCBM  correlates  to  NAV AIR’s 
FRR,  assessing  readiness  of  the  program  to  move  into  flight  test  for  score. 

The  culmination  of  the  implementation  phase  happens  at  the  final  TCBM.  The  FAA 
will  review  all  data  at  this  TCBM,  and  if  compliant  to  the  regulations,  the  FAA  issues  the 
TC  and  accompanying  TCDS.  Issuance  of  these  documents  is  the  goal  of  the  TC  process. 
This  evaluation  and  release  of  the  goal  product  correlate  to  NAV AIR’s  FCA  and  SVR, 
assessing  final  compliance  of  the  program  as  a  whole,  but  remember  NAV AIR  assesses 
more  than  airworthiness  as  the  acquiring  customer.  Note  NAV AIR’s  PRR  is  on  the  heels 
of  the  FCA  and  SVR,  supporting  the  Milestone  C  decision  to  go  into  the  production  phase, 
discussed  next. 

d.  Post  Certification  Activities  Compared  to  Production  and  Deployment, 
Operations  and  Support  Phases 

Figure  16  focuses  on  the  FAA’ s  post  certification  phase,  compared  to  the  remainder 
of  the  NAV AIR  phases:  production  and  deployment,  and  operations  and  support.  The 
immediate  product  is  the  type  inspection  report  (TIR).  The  TIR  formally  documents  the 
full  details  of  compliance  results  after  the  issuance  of  the  TC  in  the  previous  phase.  This 
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document  also  reports  on  conformity  findings,  helpful  to  NAV AIR’s  Physical 
Configuration  Audit  (PC A). 
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Figure  16.  FAA  Key  Documents  for  Post  Certification  Phase 


NAV AIR  is  the  procuring  activity  as  well  as  supporting  user  activity;  production 
execution  and  deployment  to  the  user  do  not  have  counterparts  in  the  FAA  TC  process, 
whose  purpose  is  solely  airworthiness  certification.  The  FAA’s  post  certification  activities 
phase  addresses  considerations  and  policies  for  maintaining  airworthiness.  NAV AIR’s 
operations  and  support  phase  addresses  these  same  considerations  and  a  great  deal  more. 

2.  TRR,  FRR,  FCA,  SVR  SETR  Entrance  Criteria  Mapping  to  FAA 
Elements 

The  previous  section  of  this  chapter  compares  the  entire  NAV AIR  SETR  timeline 
to  the  FAA  TC  process.  This  section  narrows  the  scope  of  the  final  analysis  to  the  TRR, 
FRR,  FCA,  and  SVR  SETR  events.  This  is  the  portion  of  the  NAV AIR  SETR  timeline 
focusing  on  verification  preparation  and  verification  completion,  and  is  the  scoped  focus 
of  this  thesis.  Mapping  diagrams  relate  FAA  elements  to  specific  SETR  entrance  criteria 
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for  these  four  events.  NAVAIR,  as  the  acquirer,  military  airworthiness  authority, 
maintenance  and  user  supporting  activity,  is  examining  many  aspects  beyond  airworthiness 
in  SETR.  These  mappings  do  not  propose  equivalency  to  entrance  criteria,  but  bring 
awareness  and  acknowledgement  of  the  FAA  elements  relevant  to  the  CDA  portions  of 
those  specific  criteria.  Acceptance  of  these  FAA  elements  as  satisfying  the  CDA  portions 
of  those  specific  criteria  is  the  true  benefit  to  executing  SETR  on  a  CDA  program. 

SETR  groups  entrance  criteria  into  three  areas:  requirements,  traceability  and 
design;  test,  evaluation  and  certification;  and  program  management  and  execution 
(NAVAIR  2015b).  These  mappings  utilize  Figure  11  to  crosscheck  consistent  mapping 
across  the  four  SETRs  with  common  entrance  criteria.  With  the  role  of  NAVAIR  as 
acquirer  and  user  differing  from  the  FAA’s  sole  focus  as  the  airworthiness  approval 
authority,  almost  no  FAA  elements  map  to  the  program  management  and  execution  group. 
There  is  one  exception  in  FCA.  To  save  graphic  space,  the  mapping  diagrams  do  not  list 
the  entrance  criteria  for  this  group  in  the  other  events.  The  Appendix  lists  all  entrance 
criteria  in  full  for  these  four  events. 

a.  TRR  Entrance  Criteria  Mapping 

Figure  17  maps  TRR  entrance  criteria  to  FAA  elements  from  the  data  generation 
sub-phase  of  the  implementation  phase.  The  FAA  and  the  applicant  hold  an  interim  TCBM 
for  certification  plan  at  the  beginning  of  the  implementation  phase,  to  finalize  certification 
planning  and  assess  readiness  for  moving  into  implementation.  Reviewing  actions  and 
results  from  this  TCBM  may  also  benefit  TRR  entrance  criteria.  Examining  a  specific 
SETR  entrance  criterion  on  the  left,  the  presence  of  a  solid  blue  line  indicates  there  is  an 
FAA  element  to  consider.  In  many  cases,  a  single  FAA  element  maps  to  several  SETR 
entrance  criteria.  Out  of  the  eleven  entrance  criteria  reviewed,  eight  have  FAA  element 
counterparts. 
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Figure  17.  TRR  Entrance  Criteria  Mapping  to  FAA  Elements 
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In  the  requirements,  traceability  and  design  group,  the  entrance  criteria  for  design 
product  baseline  (1A),  technical  interfaces  (IB),  and  critical  safety  item  (CSI)  testing  done 
(2F)  from  the  second  group  connect  to  conformity  inspections.  A  conformity  inspection 
verifies  a  product’s  build  matches  its  design,  and  is  required  for  key  components  and  sub¬ 
systems.  Any  conformity  inspections  conducted  prior  to  the  TRR  assessment  verify  those 
specific  components’  product  baseline  and  interfaces.  As  previously  discussed,  the  FAA 
accepts  safety  risk  differently  than  NAVAIR,  but  any  components  requiring  conformity 
inspections  will  categorize  as  CSIs.  The  entrance  criteria  for  traceability  to  verification 
plan  (1C)  and  safety  hazard  analysis  (ID)  connect  to  PSCPs.  These  plans  define  the  details 
of  what  regulations  are  applicable,  how  the  design  shows  satisfaction  to  those  regulations, 
and  the  verification  methodology  to  show  compliance.  The  PSCPs  provide  traceability 
from  the  parent  regulations,  or  requirements,  to  the  design  execution  and  verification 
methodology.  The  FAA’s  focus  on  safety  and  impact  to  airworthiness  requires  execution 
of  safety  hazard  analyses  to  determine  what  level  of  testing  is  required;  PSCP  development 
builds  a  safety  hazard  analysis  into  the  process. 

In  the  test,  evaluation  and  certification  group,  the  entrance  criterion  for  test  plans 
done  (2A)  connects  to  test  plans  approved.  The  FAA  requires  test  plan  approval  prior  to 
execution  by  the  applicant,  even  in  the  data  generation  phase  where  the  FAA  may  only 
witness  portions  of  testing  and  the  majority  will  not  be  for  score.  The  entrance  criteria  for 
certifications  and  flight  approvals  obtained  (2E)  map  to  experimental  airworthiness 
certificate.  When  the  applicant  needs  to  conduct  system-level  flight  tests,  the  FAA  issues 
an  experimental  airworthiness  certificate;  NAVAIR  issues  an  IFC.  The  FAA  experimental 
airworthiness  certificate  very  tightly  defines  and  limits  allowable  flights.  The  entrance 
criterion  for  CSI  testing  done  (2F)  connects  to  conformity  inspections,  already  discussed. 
The  entrance  criterion  for  design  analysis  (2G)  connects  to  design  analysis.  The  FAA  TC 
process  calls  for  design  analysis  in  the  same  manner  as  NAVAIR.  Assessment  of  design 
analysis  either  verifies  the  design  or  supports  its  readiness  to  move  into  testing. 

The  remaining  entrance  criteria  for  test  procedures,  data  collection,  discrepancy 
reporting  (2B),  facilities  (2C),  and  simulation  lab  (SIL)  verification  and  validation  (V&V) 
(2D)  do  not  benefit  from  FAA  element  mapping.  These  criteria  are  also  necessary  to 
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program  execution  by  the  applicant,  but  the  TC  process  does  not  explicitly  identify  them. 
These  test  execution  details  are  still  pertinent  to  testing  conducted  by  NAVAIR. 

b.  FRR  Entrance  Criteria  Mapping 

Figure  18  maps  FRR  entrance  criteria  to  FAA  elements  from  the  finding  sub-phase 
of  the  implementation  phase.  Comparing  FRR  to  TRR,  many  entrance  criteria  are  the  same. 
FRR  focuses  narrowly  on  readiness  to  support  flight  test,  while  TRR  assesses  readiness  to 
test  in  general.  As  a  result,  many  mapping  correlations  are  the  same.  The  FAA  and 
applicant  hold  a  pre -flight  TCBM  prior  to  authorizing  flight  tests  for  score.  Reviewing 
actions  and  results  from  this  TCBM  may  also  benefit  FRR  entrance  criteria.  Out  of  the  ten 
entrance  criteria  reviewed,  seven  have  FAA  element  counterparts 
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Figure  18.  FRR  Entrance  Criteria  Mapping  to  FAA  Elements 


In  the  requirements,  traceability  and  design  group,  the  entrance  criteria  for  design 
product  baseline  (1A),  CSI  testing  done  (2G),  and  design  tested  to  functional  requirements 
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(2H)  connect  to  conformity  inspections.  The  same  rationale  from  the  TRR  mapping  applies 
here.  The  entrance  criterion  for  design  product  baseline  (1A)  also  connects  to  the  flight- 
test-conformity  inspection.  This  conformity  inspection  verifies  the  aircraft  as  a  whole  is 
ready  to  move  into  flight  test  for  score  by  the  FAA. 

In  the  test,  evaluation  and  certification  group,  the  entrance  criterion  for  test  plans 
done  (2A)  connects  to  test  plans  approved.  The  same  rationale  from  the  TRR  mapping 
applies  here.  The  entrance  criteria  for  certifications  obtained  (2E)  and  flight  clearance 
obtained  (2F)  map  to  experimental  airworthiness  certificate  and  TIA.  FRR  splits  the  TRR 
entrance  criterion  for  “certifications  and  flight  approvals  obtained”  into  two  separate 
criteria.  The  rationale  from  the  TRR  mapping  for  the  experimental  airworthiness  certificate 
applies  here,  mapping  the  experimental  airworthiness  certificate  to  both  2E  and  2F.  The 
addition  here  in  FRR  is  also  mapping  2E  and  2F  to  the  TIA.  The  FAA  issues  a  TIA  to 
authorize  the  flight-test-conformity  inspection  and  flight  tests  for  score.  The  entrance 
criterion  for  CSI  testing  done  (2G)  connects  to  conformity  inspections,  already  discussed. 
The  same  rationale  from  the  TRR  mapping  applies  here. 

The  entrance  criterion  for  flight  tests  to  support  modeling  and  simulation  (M&S) 
validation  identified  (2B)  is  not  explicitly  addressed  in  the  FAA  TC  process.  Execution 
details  and  related  processes  may  address,  but  further  investigation  is  required.  The 
remaining  entrance  criteria  for  test  procedures,  data  collection,  discrepancy  reporting  (2C), 
and  facilities  (2D)  do  not  benefit  from  FAA  element  mapping.  These  test  execution  details 
are  still  pertinent  to  testing  conducted  by  NAVAIR. 

c.  FCA  Entrance  Criteria  Mapping 

Figure  19  maps  FCA  entrance  criteria  to  FAA  elements  from  the  finding  sub-phase 
of  the  implementation  phase.  FCA  is  an  audit  rather  than  a  SETR  event,  explaining  its 
reduced  entrance  criteria.  The  FAA  and  applicant  hold  a  final  TCBM  at  the  end  of  the 
implementation  phase.  Reviewing  actions  and  results  from  this  TCBM  may  also  benefit 
FCA  entrance  criteria.  Out  of  the  five  entrance  criteria  reviewed,  four  have  FAA  element 
counterparts. 
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Figure  19.  FCA  Entrance  Criteria  Mapping  to  FAA  Elements 


In  the  system  requirements  and  capabilities  group,  the  entrance  criteria  for  audit 
physical  system  reflects  functional  baseline  (1A)  and  traceability  to  verification  plan  (IB) 
connect  to  the  TIR.  Generation  of  this  report  actually  occurs  in  the  post-certification 
activities  phase,  documenting  all  the  results  and  details  justifying  the  release  of  the  TC.  If 
a  draft  is  available,  or  execution  timing  aligns  with  the  FCA  audit,  the  TIR  supports  these 
criteria.  As  shown  in  Figure  19,  the  traceability  to  verification  plan  (IB)  entrance  criterion 
is  also  in  TRR  and  FRR.  The  mapping  is  different  here  as  the  artifacts  discussed  shift  from 
pre-verification  to  post-verification. 

In  the  test,  evaluation  and  certification  group,  the  entrance  criterion  for  data 
artifacts  show  compliance  (2A)  connects  to  the  TC  and  TCDS.  These  products  are  the  FAA 
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airworthiness  certificate  and  issuance  does  not  occur  without  full  compliance  proven  and 
accepted. 

In  the  program  management  and  execution  sub-group,  the  entrance  criterion  for 
configuration  management  procedures  (2C)  connects  to  the  conformity  plan.  Generation 
of  this  plan  occurs  earlier  in  the  compliance-planning  phase,  but  supports  this  entrance 
criterion  by  documenting  the  applicant’s  configuration  management  processes  in  addition 
to  conformity  planning  execution. 

d.  SVR  Entrance  Criteria  Mapping 

Figure  20  maps  SVR  entrance  criteria  to  FAA  elements  from  the  finding  sub-phase 
of  the  implementation  phase.  SVR  and  FCA  often  occur  in  conjunction.  The  FAA  and 
applicant  hold  a  final  TCBM  at  the  end  of  the  implementation  phase.  Reviewing  actions 
and  results  from  this  TCBM  may  also  benefit  SVR  entrance  criteria.  Out  of  the  four 
entrance  criteria  reviewed,  three  have  FAA  element  counterparts. 
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Figure  20.  SVR  Entrance  Criteria  Mapping  to  FAA  Elements 


In  the  system  requirements  and  capabilities  group,  the  entrance  criterion 
traceability  to  verification  plan  connects  to  the  TIR.  As  discussed  in  the  FCA  mapping, 
generation  of  this  report  occurs  in  the  post-certification  activities  phase.  If  a  draft  is 
available,  or  execution  timing  aligns  with  SVR,  the  TIR  supports  this  criterion. 

In  the  test,  evaluation  and  certification  group,  the  entrance  criterion  for 
certifications  for  fielding  (2B)  connects  to  the  TC,  TCDS,  and  instructions  for  continued 
airworthiness.  These  products  are  the  FAA  airworthiness  certificate  and  accompanying 
approved  maintenance  manuals.  The  aircraft  must  have  these  for  flight  and  operations.  The 
entrance  criterion  for  preliminary  NATOPS,  also  known  as  the  PFC,  (2C)  connects  to  the 
flight  manual.  This  is  also  FAA  approved  and  required  for  release  of  the  TC  and  TCDS. 
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The  remaining  entrance  criterion  for  current  test  results  and  remaining  tests  indicate 
acceptable  risk  (2A)  does  not  have  an  FAA  element  counterpart.  The  FAA  makes  no 
allowance  for  uncompleted  testing  and  acceptance  of  risk;  everything  must  be  complete 
and  compliant  to  gain  release  of  a  TC. 

D.  CHAPTER  SUMMARY 

The  analyses  presented  in  this  chapter  started  with  a  comparison  of  airworthiness 
definitions  and  airworthiness  tenets,  identifying  similarities  and  differences.  This  justifies 
the  NAVAIR  approach  to  accept  FAA  airworthiness  certifications,  and  the  assessment 
NAVAIR  has  a  tighter  airworthiness  scope  due  to  increased  acceptance  of  safety  risk.  The 
chapter  then  provided  an  internal  comparison  of  TRR,  FRR,  FCA,  and  SVR  entrance 
criteria  used  in  the  final  analysis  as  a  crosscheck.  The  largest  analysis  effort  compared  the 
NAVAIR  SETR  timeline  to  the  FAA  TC  process,  justifying  the  proposed  alignment  of  the 
two.  Examining  each  phase  of  the  timeline  comparison,  allows  more  detailed  analysis.  The 
final  analysis  used  the  timeline  comparisons  to  map  FAA  elements  to  specific  SETR  event 
entrance  criteria.  Four  mapping  diagrams,  for  TRR,  FRR,  FCA,  and  SVR,  are  the  product 
of  this  chapter. 
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IV.  CASE  STUDY 


This  chapter  discusses  a  case  study,  the  Presidential  Helicopter  Replacement 
Program,  providing  a  background,  presenting  pertinent  challenges,  lessons  learned, 
applying  the  analyses  of  this  thesis,  and  assessing  the  benefits  of  the  analyses. 

A.  PRESIDENTIAL  HELICOPTER  REPLACEMENT  PROGRAM 

A  high-level  background  of  the  VH-92A  program  provides  information  justifying 
its  application  as  a  case  study  for  this  thesis.  Discussion  of  some  challenges  and  lessons 
learned  during  the  beginning  of  this  program’s  execution  give  a  real-world  application  of 
several  key  points  in  thesis,  and  lead  into  the  following  analysis  discussion. 

1.  Background 

Today,  the  United  States  Marine  Corps  operates  two  helicopter  models  to  fly  the 
President  of  the  United  States  of  America  (NAVAIR  2016b).  The  VH-3D  and  the  VH- 
60N,  beginning  military  service  in  the  1970s  and  1980s,  respectively,  are  both  classic 
military  acquisition  and  military  certified  helicopters.  As  these  aircraft  age,  a  replacement 
helicopter  is  needed.  The  original  intended  replacement,  the  VH-71,  terminated  due  to  cost 
overgrowth  (Gertler  2009).  The  need  remained,  and  a  new  approach  began.  In  2014,  a 
contract  was  awarded  to  Sikorsky  Aircraft  Corporation  to  integrate  defined  government 
mission  systems  into  the  commercially  available  S-92A  aircraft;  the  final  integrated  aircraft 
is  designated  the  VH-92A  (NAVAIR  2016b).  A  key  point  of  this  acquisition  strategy  is 
maintaining  the  existing  airworthiness  certification.  The  original  airworthiness 
certification  authority,  being  the  FAA  for  the  S-92A,  is  to  provide  the  airworthiness 
certification  for  the  VH-92A  as  well,  to  the  maximum  extent  possible.  This  is  a  deliberate 
decision  to  modify  an  already  existing,  airworthiness-certified  aircraft  and  to  utilize  the 
same  airworthiness  authority  for  the  military  modifications.  Doing  so  controls  cost,  risk, 
and  acquisition  time,  while  still  meeting  the  performance  requirements  of  the  mission. 
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2.  Challenges  and  Lessons  Learned 

The  approach  of  using  the  current  aircraft’s  airworthiness  authority  to  the  maximum 
extent  possible  for  the  military  modifications  is  atypical  of  military  acquisition.  While  AIR- 
4.0P  acknowledges  and  accepts  FAA  airworthiness  certifications  (Airworthiness  and 
CYBERSAFE  Directorate  [AIR-4.0P]  2016),  deliberate  use  of  the  FAA  to  assess  military 
modifications  is  unfamiliar  territory  for  most  of  NAVAIR.  NAVAIR  is  assessing  only 
those  airworthiness  areas  the  FAA  cannot  or  will  not,  due  to  lack  of  or  explicit  conflict 
with  civil  regulations.  Refreshers  of  this  fundamental  premise  in  the  VH-92A’s  acquisition 
strategy  are  required  regularly.  Lack  of  familiarity  and  natural  turnover  in  personnel  require 
education  on  FAA  processes  on  a  frequent  basis.  The  program  frequently  corrects 
assumptions  of  a  return  to  full  NAVAIR  airworthiness  certification  as  the  acquisition 
program  continues.  The  perception  of  a  loss  of  control  requires  re-assurance. 

The  difference  between  airworthiness  and  performance  assessments  affirms 
NAVAIR  authorities’  role  in  this  program’s  acquisition.  While  the  FAA  is  assessing 
airworthiness  in  the  vast  majority,  NAVAIR  is  still  assessing  all  requirements  for 
verification  of  performance.  The  challenge  here  is  to  understand  how  the  FAA 
airworthiness  certification  process  assesses  any  given  requirement  in  order  to  take 
advantage  by  not  repeating  effort.  Full  awareness,  understanding,  and  engagement  by 
NAVAIR  during  this  process  maximizes  program  efficiency,  allowing  acceptance  of 
performance  requirements  during  the  FAA  airworthiness  certification  process  and  reducing 
test  efforts  for  performance  verification. 

Limiting  SETR  scope  to  only  integration  of  new  and  modified  design  efforts 
reinforces  the  premise  of  accepting  the  existing  aircraft  design  capabilities  and  its  original 
airworthiness  assessment.  The  performance  of  the  modified  aircraft  as  a  whole  is  still 
paramount.  Integration  impacts  are  fully  assessed,  but  without  re-evaluation  of  the  baseline 
S-92A  against  NAVAIR  standards. 

Requirements  verification  planning  identifies  what  efforts,  analyses,  tests,  and 
documents  are  necessary.  Identification  of  when  these  products  are  available  during 
execution  of  the  program  leads  to  a  timeline  and  the  representative  S-curve  shown  in  Figure 
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21.  Notice  the  curve  begins  at  sixty;  the  baseline  S-92A  aircraft  itself  satisfies  some  of  the 
VH-92A  system  level  requirements.  The  majority  of  requirements  are  verifiable  by  the 
time  of  the  FAA  TC  release,  456  out  of  677,  or  67%.  The  TC  release  also  marks  the  end  of 
the  contractor  test  (CT)  period.  The  orange  area  identifies  these  verifiable  requirements  by 
the  time  of  the  TC  release;  this  encompasses  all  verification  methodologies,  inclusive  of 
analysis,  inspection,  demo,  and  test.  The  blue  area  identifies  the  remainder  of  requirements 
verification,  beginning  post-TC  with  the  government  integration  test  (IT)  period. 
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Figure  21 .  Requirement  Verification  over  Time 

Recalling  Figure  2,  NAVAIR  versus  FAA  requirements  Venn  diagram,  Figure  22 
draws  a  correlation  to  Figure  21  with  matching  colors.  Between  the  normal  execution  of 
the  program  and  the  deliberate  efforts  to  achieve  an  FAA  TC,  many  requirements  are 
verifiable  by  the  time  of  the  TC  release;  the  orange  area  of  requirements  verifiable  prior  to 
TC  relates  to  the  orange  area  of  the  FAA  airworthiness  set.  The  blue  post-TC  area  relates 
to  the  blue  areas  of  the  NAVAIR  airworthiness  and  performance  sets  that  protrude  outside 
the  FAA  airworthiness  set,  or  the  complement  of  those  sets  with  the  FAA  airworthiness 
set. 
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Figure  22.  NAVAIR  vs.  FAA  Venn  Diagram,  Pre-  and  Post-TC 

In  a  typical  military  acquisition  program,  CT  does  not  have  the  goal  of  achieving  a 
civil  airworthiness  certificate,  an  FAA  TC,  prior  to  delivery  to  the  government.  The  TC 
line  of  Figure  21  is  then  no  longer  a  TC  line,  becoming  instead  a  contractual  definition 
delineating  when  CT  ends  and  IT  begins.  This  line  shifts  to  the  left,  and  typically,  IT 
conducts  a  large  amount  of  requirement  verification  effort.  In  the  VH-92A  program, 
Sikorsky  will  be  verifying  the  majority  of  requirements,  67%,  as  a  side  benefit  of  pursing 
the  FAA  TC.  Understanding  execution  details  is  critical  to  maximizing  efficiencies  and  the 
ability  to  scope  government  IT  to  the  minimum  required  effort. 

This  also  leads  to  the  challenge  of  assessing  TRR  and  FRR  at  the  appropriate  time. 
TRR  is  held  as  needed  for  incremental  and  system  level  test  preparation.  System-level  TRR 
and  FRR,  prior  to  the  beginning  of  government  IT,  support  requirement  verification  efforts. 
In  the  VH-92A  program,  understanding  the  test  efforts  conducted  during  CT  in  support  of 
the  FAA  TC  release  and  what  the  TRR  and  FRR  SETR  events  evaluate  requires  a  paradigm 
shift,  evaluating  appropriate  content  during  CT  and  prior  to  the  beginning  of  FT. 

B.  APPLICATION  OF  THESIS 

This  section  re-evaluates  the  SETR  timeline  to  FAA  TC  process  comparison  for  the 
VH-92A  program.  After  checking  this  analysis  for  validity  with  the  VH-92A  program,  this 
section  then  discusses  the  mapping  diagrams  for  TRR,  FRR,  FCA,  and  SVR.  The  author 
then  discusses  the  benefits  of  this  thesis  work  for  the  VH-92A  program. 
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1.  SETR  Timeline  to  FA  A  TC  Process  for  the  VH-92A  Program 

The  VH-92A  program  entered  post  Milestone  B,  with  the  release  of  the  EMD 
contract  in  2014  (NAVAIR  2016b).  This  leads  to  a  shift  in  the  alignment  of  the  two 
timelines;  the  entire  FAA  TC  process  now  executes  during  the  EMD  phase,  see  Figure  23. 
The  SETR  events  also  shift  to  align  with  contract  award  execution.  The  FAA  TC  process 
must  complete  prior  to  the  completion  of  the  EMD  phase  to  allow  for  government  IT 
execution  and  full  assessment  of  the  entire  program  at  FCA  and  SVR,  in  support  of 
Milestone  C. 

Notice  the  pre-flight  TCBM  still  correlates  to  FRR  in  its  purpose,  but  the  program- 
level  ERR  must  now  align  with  the  end  of  the  FAA  TC  process,  assessing  the  program’s 
readiness  to  move  into  government  IT.  Remember  TRRs  occur  as  needed,  prior  to  testing 
in  the  data  generation  phase  and  often  held  in  conjunction  with  FRR  prior  to  IT. 
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While  the  FAA  TC  process  phase  counterparts  on  the  NAVAIR  acquisition  timeline 
remain  the  same  for  reasons  previously  discussed,  the  goal  of  this  analysis  is  to  map  FAA 
TC  process  phases  to  SETR  events.  Since  the  SETR  events  have  also  shifted,  the  FAA 
element  mapping  to  SETR  events  is  unchanged. 

2.  TRR,  FRR,  FCA,  SVR  SETR  Entrance  Criteria  Mapping  to  FAA 
Elements  for  VH-92A 

The  diagrams  mapping  of  TRR,  FRR,  FCA,  and  SVR  entrance  criteria  to  FAA 
elements  apply  unchanged  to  this  program.  The  challenge  will  be  to  obtain  access  to  the 
referenced  documents  within  the  bounds  of  the  contract  since  the  FAA  cannot  provide  them 
directly,  and  assess  missed  opportunities  for  efficiencies  where  there  are  gaps. 

3.  Benefits  of  the  Analyses  for  the  VH-92A 

The  timeline  comparison  provides  a  tool  for  education  of  FAA  processes  and 
alignment  to  NAVAIR  acquisition  and  SETR  for  this  program.  Awareness  and 
understanding  of  the  differences  and  similarities  is  important  to  maximize  further  the 
benefits  of  executing  a  CDA  program. 

The  high  percentage  of  requirements  verified  during  execution  of  the  FAA  TC 
process,  67%,  is  proof  of  execution  efficiency  opportunity.  The  identification  of  FAA 
elements  mapped  to  SETR  entrance  criteria  provides  great  potential  in  reducing  evaluation 
effort,  especially  given  the  high  requirement  verification  percentage.  Understanding  the 
work  undergone  in  the  FAA  TC  process  allows  the  government  to  be  aware  of  and  engaged 
prior  to  its  execution,  gaining  efficiencies  by  avoiding  duplication  of  that  work  later,  saving 
test  efforts  and  therefore  saving  time  and  cost.  The  amount  of  savings  is  unquantifiable,  as 
programmatic  pressures  and  the  already-defined  contract  may  reduce  the  maximum 
potential  benefits  during  execution,  but  the  potential  for  avoidance  is  clear.  The  execution 
of  TRR  and  FRR  will  benefit  by  this  understanding  and  involvement.  FCA  and  SVR  can 
leverage  a  great  deal  of  requirements  verification  executed  by  the  FAA.  By  knowing  what 
documents  to  review,  for  a  given  entrance  criteria,  the  government  can  accept  efforts 
already  performed  and  documented  for  the  FAA. 
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C.  CHAPTER  SUMMARY 

The  Presidential  Helicopter  Replacement  Program  is  a  relevant  case  study  for  the 
focus  of  this  thesis,  a  CDA  program  implementing  SETR.  The  basics  of  the  program’s 
history  and  the  VH-92A’s  acquisition  strategy  lead  to  a  discussion  on  some  challenges  and 
lessons  learned  in  the  early  stages  of  execution.  The  majority  of  requirements  for  this 
program,  67%,  are  verifiable  by  the  time  of  the  FAA  TC  release.  The  FAA  elements  to 
SETR  entrance  criteria  mapping  remains  unchanged,  verifying  their  applicability  and 
benefits  to  this  program.  The  high  requirement  verification  percentage  ensures  the  SETR 
entrance  criteria  mapping  provides  great  potential  in  reducing  the  effort  of  evaluation. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


This  chapter  draws  conclusions,  reviews  the  research  questions,  and  provides 
recommendations  for  further  work. 

A.  CONCLUSIONS 

While  the  FAA  elements  cannot  completely  satisfy  any  given  NAVAIR  SETR 
entrance  criteria  due  to  NAVAIR  SETR  assessing  a  much  broader  scope,  there  is  no 
question  of  benefit.  For  TRR,  eight  out  of  eleven  entrance  criteria  mapped  to  FAA 
elements;  for  ERR,  seven  out  of  ten;  for  FCA,  four  out  of  five;  and  for  SVR,  three  out  of 
four.  See  Figure  24.  Out  of  30  evaluated  SETR  entrance  criteria,  22  map  to  FAA  elements; 
this  is  over  73%.  This  high  ratio  proves  there  is  benefit  in  accepting  FAA  processes  and 
documentation  for  military  CDA  programs,  even  with  FAA  elements  providing  partial 
satisfaction  to  the  SETR  entrance  criteria. 
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Figure  24.  Count  of  TRR,  FRR,  FCA,  SVR  Entrance  Criteria  Mapped  to  FAA  Elements 

Although  cost  and  time  to  execute  savings  cannot  be  quantified,  it  is  clear  to  see 
there  is  room  for  savings  due  to  the  high  number  of  SETR  entrance  criteria  mapped.  No 
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two  programs  are  the  same,  and  each  may  have  different  amounts  of  performance 
requirement  verification  outside  of  the  FAA  airworthiness  boundary.  In  the  Presidential 
Helicopter  Replacement  Program  case  study  of  Chapter  IV,  approximately  67%  of 
requirements  will  be  validated  thru  the  FAA  TC  process,  ensuring  benefits  from  the  SETR 
entrance  criteria  mapping,  if  program  execution  allows. 

Identification  and  referencing  FAA  processes  is  very  helpful  to  NAVAIR  education 
and  recognition  of  efficiencies  in  a  military  CDA  program.  Training  would  improve  the 
likelihood  of  implementing  these  benefits.  Adding  references  within  NAVAIR  instructions 
and  acknowledging  specific  FAA  elements  would  support  acceptance  of  commercial 
processes.  This  also  would  assist  in  knowing  what  deliverables  to  request  when  writing  a 
contract,  since  the  FAA  cannot  provide  anything  direct  to  the  government. 

B.  REVIEW  OF  RESEARCH  QUESTIONS 

The  goal  of  this  thesis  is  to  create  a  mapping  of  FAA  process  and  documentation  to 
NAVAIR  SETR  events  and  their  event  criteria.  This  enables  military  CDA  programs  to 
reduce  SETR  and  program  execution  effort  by  acknowledging  and  accepting  the  FAA 
elements  instead  of  requiring  duplicative  effort  to  NAVAIR  standards.  Recalling  the 
research  questions  of  Chapter  I: 

•  What  is  the  benefit  to  NAVAIR  SETR  execution  on  CDA  programs  by 
considering  FAA  processes  and  documentation? 

•  For  requirements  verification  planning,  how  do  FAA  processes  and 
documentation  for  test  readiness  and  flight  readiness  map  to  NAVAIR 
SETR  Test  Readiness  Review  (TRR)  and  Flight  Readiness  Review  (FRR) 
entrance  criteria? 

•  For  requirements  verification  acceptance,  how  do  FAA  performance 
evaluations  map  to  NAVAIR  SETR  Functional  Configuration  Audit  (FCA) 
and  System  Verification  Review  (SVR)  entrance  criteria? 

This  thesis  achieved  its  goal  by  comparing  the  NAVAIR  SETR  timeline  to  the  FAA 
TC  process  at  a  high  level,  and  then  by  mapping  FAA  elements  to  TRR,  FRR,  FCA,  and 
SVR  entrance  criteria  at  a  detail  level.  These  four  SETR  events  focus  on  requirement 
verification  planning  and  evaluation,  where  the  majority  of  the  FAA  TC  process  effort 

aligns  with  NAVAIR  SETR  efforts.  This  thesis  answers  these  research  questions  with  the 

60 


SETR  entrance  criteria  to  FAA  element  mapping  diagrams,  and  demonstrates  there  is  a 
significant  savings  potential  by  doing  so  in  the  Presidential  Helicopter  case  study. 

C.  RECOMMENDATIONS 

As  discussed  in  Chapter  I,  the  NAVAIR  SETR  process  does  not  acknowledge 
commercial  documentation  generated  in  pursuit  of  a  civil  airworthiness  certificate  for 
military  CDA  programs.  This  is  a  missed  opportunity,  an  additional  benefit  to  the  reduced 
costs  and  execution  time  of  using  commercial  products  and  processes  in  a  military 
acquisition.  The  NAVAIR  airworthiness  authority  accepts  FAA  airworthiness  products 
without  further  review,  but  the  NAVAIR  SETR  process  does  not  acknowledge  FAA 
products  in  any  manner. 

Update  and  reinstate  NAVAIRINST  13100.15  to  renew  NAVAIR’ s 
acknowledgement  of  commercial  processes  in  SETR  evaluations  on  CDA  programs, 
potentially  in  the  manner  of  this  thesis.  Avoid  the  use  of  “program  points,”  which  are  not 
used  by  the  FAA  or  NAVAIR.  Ensure  proper  references  to  FAA  documentation  to  enable 
education  of  commercial  process  details.  Link  to  this  instruction  from  within 
NAVAIRINST  4355.19  for  SETR  execution,  or  incorporate  within  NAVAIRINST 
4355.19  itself.  This  would  provide  awareness  of  commercial  terms,  documentation,  and 
processes  for  use  in  CDA  program  execution. 

Cross-reference  MIL-HDBK-516,  explicitly  for  airworthiness  certification  criteria, 
to  the  JSSGs,  for  performance  requirement  development.  These  two  documents  do  not 
reference  each  other,  but  NAVAIR  uses  both  for  requirement  development.  Providing  a 
unified  document  would  consolidate  potential  conflicts,  redundancies,  and  could  identify 
a  requirement’s  purpose  as  airworthiness  only,  performance  only,  or  performance  affecting 
airworthiness.  This  would  provide  clarity  when  NAVAIR  is  planning  for  airworthiness 
assessments  and  scoping  test  efforts.  An  additional  recommendation  is  to  map  also  the 
MIL-HDBK-516  and  JSSG  requirements  to  those  of  the  14  CFR.  This  would  enable 
potential  alignment  of  the  military  CDA  program  requirements  to  14  CFR  requirements 
during  requirement  development;  this  would  also  enable  benefit  military  CDA  programs 
in  knowing  specifically  which  requirements  the  FAA  would  assess  for  a  TC. 
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There  are  additional  SETR  events  that  could  also  benefit  from  mapping  FAA 
processes  and  documentation,  as  identified  in  the  timeline  analysis  of  Chapter  III. 
Completing  the  entrance  criteria  mapping  effort  for  SRR-II,  SFR,  PDR,  CDR,  and  PCA 
would  identify  similar  benefits  to  those  events  as  well. 
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APPENDIX.  4355.19  TRR,  FRR,  FCA,  SYR  ENTRANCE  CRITERIA 


NAVAIRINST  4355.19  contains  the  full  entrance  criteria  for  all  SETRs.  This 
appendix  provides  a  ready  listing  of  the  entrance  criteria  for  the  four  SETRs  discussed  in 
this  thesis  (NAVAIR  2015b). 

A.  TRR  ENTRANCE  CRITERIA 

1.  System  Requirements,  Traceability  and  Design 

a.  The  current  design  reflects  the  product  baseline. 

b.  System  technical  interfaces  (including  external  interfaces)  have  been 
updated  and  documented 

c.  Traceability  from  design  requirements  to  verification  plan  is  complete 
and  has  been  updated 

d.  Subsystem  level  safety  and  hazard  analysis  supports  the  product  baseline 

2.  Test,  Evaluation  and  Certification  of  Product 

a.  Test  plans  are  ready  for  final  review 

b.  Test  procedures  are  updated  and  address  Acceptance  Test  Procedures 
(ATP);  data  collection,  reduction,  analysis;  test  and  deficiency  reporting 

c.  Coordination  of  test  facilities  is  complete  and  facilities  are  available  for  testing 

d.  SIL  V&V  completed,  results  are  documented,  and  the  SIL  is  under  CM  control 

e.  Certifications  and/or  flight  approvals  have  been  obtained 

f.  CSI  and  CSA  testing  is  complete  and  documented 

g.  Analysis  of  the  current  design  verifies  conformance  to  the  product  baseline 

3.  Project  Management  and  Execution  (Planning,  Assessment  and  Control) 

a.  Maturity  of  the  CTEs  are  in  accordance  with  the  TMP 

b.  Planning  for  IPS  elements  is  current  and  supports  the  baseline 

c.  Earned  Value  data  reflects  current  program  status 

d.  CARD  and/or  cost  estimates  are  up  to  date 

e.  The  IMS  with  critical  path  reflects  current  program  status 

f.  Program  execution  risks  (programmatic  and  technical)  are  identified,  and 
mitigation  plans  in  place 

B.  FRR  ENTRANCE  CRITERIA 

1.  System  Requirements,  Traceability  and  Design 

a.  The  current  design  reflects  the  product  baseline 

b.  Traceability  from  design  requirements  to  verification  plan  is  complete 
and  has  been  updated 

2.  Test,  Evaluation  and  Certification  of  Product 

a.  Test  plans  have  been  drafted,  are  in  the  approval  cycle,  and  ready  for 
Executive  Review  Board 

b.  Flight  test  requirements  supporting  M&S  validation  are  identified 


63 


c.  Test  procedures  have  been  updated  and  address  ATP,  data  collection, 
reduction,  analysis,  test  and  deficiency  reporting 

d.  Coordination  of  test  facilities  is  complete  and  facilities  are  available  for 

testing 

e.  Certifications  required  for  flight  testing  are  in  place 

f.  Interim  Flight  Clearance  to  support  flight  test  is  in  place 

g.  CSI  and  CSA  testing  is  complete  and  documented 

h.  The  design  has  been  tested  against  the  functional  requirements 
3.  Project  Management  and  Execution  (Planning,  Assessment  and  Control) 

a.  Maturity  of  the  CTE  are  in  accordance  with  the  TMP 

b.  Planning  for  the  IPS  elements  is  current  and  supports  the  baseline 

c.  Earned  Value  data  reflects  current  program  status 

d.  CARD  and/or  cost  estimates  are  up  to  date 

e.  The  IMS  with  critical  path  reflects  current  program  status 

f.  Program  execution  risks  (programmatic  and  technical)  are  identified,  and 
mitigation  plans  in  place 

g.  CDRL  and  data  requirements  are  up  to  date 

C.  FCA  ENTRANCE  CRITERIA 

1.  System  Requirements  and  Capabilities 

a.  The  physical  system  reflects  the  functional  baseline 

b.  Traceability  from  design  and  functional  requirements  to  verification  plan 
is  complete  and  has  been  updated 

2.  Test,  Evaluation  and  Certification  of  Product 

a.  Engineering  data  artifacts  have  been  reviewed  and  demonstrates 
compliance  with  the  functional  baseline 

b.  Project  Management  and  Execution  (Planning,  Assessment  and  Control) 

c.  CM  procedures  are  up  to  date  and  address  program  needs 

D.  SVR  ENTRANCE  CRITERIA 

1.  System  Requirements  and  Capabilities 

Traceability  from  design  and  functional  requirements  to  verification  plan  is 
complete  and  has  been  updated 

2.  Test,  Evaluation  and  Certification  of  Product 

a.  Current  test  Results  and  remaining  tests  planned  have  an  acceptable  level 

of  risk 

b.  Certifications  required  for  fielding  are  in  place  or  will  be  in  place  by 
Initial  Operation  Capability 

c.  Permanent  Flight  Clearance  (PFC)  in  the  form  of  Preliminary  Naval 
Aviation  Training  and  Operating  Procedures  Standardization  (NATOPS)  and  Naval 
Aviation  Technical  Information  Product  (NATIP)  (or  equivalent)  to  support 
operational  flight  test  is  in  place 

3.  Project  Management  and  Execution  (Planning,  Assessment  and  Control) 
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a.  The  Risk  Management  process  is  operating  across  the  program,  including 
contractor  and  government  activities 

b.  CSI  and  CSA  are  identified,  documented  and  are  being  managed 

c.  Planning  for  IPS  elements  is  current  and  supports  the  baseline 

d.  CDRL  and  data  requirements  are  up  to  date 

e.  Product  baseline  is  producible  as  defined  by  Manufacturing,  Producibility 
and  Quality  requirements  as  verified  by  the  results  of  the  IPRR 
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